[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:21 dist-test-slave-rc1w krb5kdc[4064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:21 dist-test-slave-rc1w krb5kdc[4064](info): set up 2 sockets
Sep 24 07:18:21 dist-test-slave-rc1w krb5kdc[4064](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 07:18:22.654301 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:22 dist-test-slave-rc1w krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:18:22.777175 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:34265
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:34265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:22.857401 4077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:22.857642 4077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:22.857703 4077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:22.858862 4077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:22.858928 4077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:22.858951 4077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:22.858973 4077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:22.859002 4077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:22.860442 4077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:34265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:34265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:22.860719 4077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:22.860951 4077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:18:22.866408 4077 server_base.cc:1034] running on GCE node
W20260924 07:18:22.866452 4082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:22.867933 4083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:22.867939 4085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:22.868315 4077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:22.868709 4077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:22.869884 4077 hybrid_clock.cc:648] HybridClock initialized: now 1790234302869854 us; error 43 us; skew 500 ppm
Sep 24 07:18:22 dist-test-slave-rc1w krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234302, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:22.873611 4077 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:18:22.874231 4077 webserver.cc:460] Webserver started at http://127.3.246.126:36081/ using document root <none> and password file <none>
I20260924 07:18:22.874495 4077 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:22.874565 4077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:22.874742 4077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:22.876094 4077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "6fd00f547e2a447dab3c08c1f961df3a"
format_stamp: "Formatted at 2026-09-24 07:18:22 on dist-test-slave-rc1w"
server_key: "f01da9d0e9459d25bee2d913155827eb"
server_key_iv: "0f7e94b05a4c2d02199513be0139a9c4"
server_key_version: "encryptionkey@0"
I20260924 07:18:22.876559 4077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "6fd00f547e2a447dab3c08c1f961df3a"
format_stamp: "Formatted at 2026-09-24 07:18:22 on dist-test-slave-rc1w"
server_key: "f01da9d0e9459d25bee2d913155827eb"
server_key_iv: "0f7e94b05a4c2d02199513be0139a9c4"
server_key_version: "encryptionkey@0"
I20260924 07:18:22.878038 4077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:18:22.878762 4092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:22.878943 4077 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:22.879031 4077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "6fd00f547e2a447dab3c08c1f961df3a"
format_stamp: "Formatted at 2026-09-24 07:18:22 on dist-test-slave-rc1w"
server_key: "f01da9d0e9459d25bee2d913155827eb"
server_key_iv: "0f7e94b05a4c2d02199513be0139a9c4"
server_key_version: "encryptionkey@0"
I20260924 07:18:22.879119 4077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:22.892059 4077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:22.894194 4077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:22.894361 4077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:22.898618 4077 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:34265
I20260924 07:18:22.898651 4144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:34265 every 8 connection(s)
I20260924 07:18:22.899024 4077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:22.899911 4145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:22.901856 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 4077
I20260924 07:18:22.901966 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:22.902309 4145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a: Bootstrap starting.
I20260924 07:18:22.902597 4057 external_mini_cluster.cc:1428] Setting key da3783fac36fb70f94c8f3393f720dc1
I20260924 07:18:22.902930 4145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:22.903218 4145 log.cc:824] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:22.904104 4145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a: No bootstrap required, opened a new log
I20260924 07:18:22.906456 4145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd00f547e2a447dab3c08c1f961df3a" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34265 } }
I20260924 07:18:22.906626 4145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:22.906663 4145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd00f547e2a447dab3c08c1f961df3a, State: Initialized, Role: FOLLOWER
I20260924 07:18:22.906805 4145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd00f547e2a447dab3c08c1f961df3a" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34265 } }
I20260924 07:18:22.906896 4145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:22.906929 4145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:22.906970 4145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:22.908087 4145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd00f547e2a447dab3c08c1f961df3a" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34265 } }
I20260924 07:18:22.908202 4145 leader_election.cc:302] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fd00f547e2a447dab3c08c1f961df3a; no voters:
I20260924 07:18:22.908367 4145 leader_election.cc:288] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:22.908435 4150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:22.908617 4145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:22.908628 4150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [term 1 LEADER]: Becoming Leader. State: Replica: 6fd00f547e2a447dab3c08c1f961df3a, State: Running, Role: LEADER
I20260924 07:18:22.908828 4150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd00f547e2a447dab3c08c1f961df3a" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34265 } }
I20260924 07:18:22.909217 4151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fd00f547e2a447dab3c08c1f961df3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd00f547e2a447dab3c08c1f961df3a" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34265 } } }
I20260924 07:18:22.909304 4151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:22.909524 4152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fd00f547e2a447dab3c08c1f961df3a. Latest consensus state: current_term: 1 leader_uuid: "6fd00f547e2a447dab3c08c1f961df3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd00f547e2a447dab3c08c1f961df3a" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34265 } } }
I20260924 07:18:22.909897 4152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:22.910033 4159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:22.910759 4159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:22.914074 4159 catalog_manager.cc:1347] Generated new cluster ID: ad57877de96b4f289ffd26c8bb550839
I20260924 07:18:22.914124 4159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:18:22 dist-test-slave-rc1w krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:22.920735 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:22.903495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.3.246.126:34265)
0924 07:18:22.903668 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:22.903687 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:22.903951 (+ 264us) server_negotiation.cc:407] Connection header received
0924 07:18:22.906434 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:22.906439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:22.906482 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:22.906590 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:22.908800 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:22.910367 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:22.911855 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:22.912238 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:22.912317 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:22.916483 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:22.916487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:22.916487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:22.916499 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:22.918740 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:22.919221 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:22.919224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:22.919225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:22.919347 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:22.919619 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:22.919620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:22.919620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:22.919935 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:22.920103 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:22.920359 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 07:18:22.920506 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260924 07:18:22.924546 4159 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:22.925024 4159 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:22.929195 4159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fd00f547e2a447dab3c08c1f961df3a: Generated new TSK 0
I20260924 07:18:22.929366 4159 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:18:22.957785 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34265
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:23.040257 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:23.040458 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:23.040484 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:23.041604 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:23.041640 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:23.041709 4171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:23.043176 4171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34265
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:23.043366 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:23.043540 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:23.048606 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:23.048605 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:23.048631 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:23.048836 4171 server_base.cc:1034] running on GCE node
I20260924 07:18:23.049100 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:23.049350 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:23.050571 4171 hybrid_clock.cc:648] HybridClock initialized: now 1790234303050534 us; error 71 us; skew 500 ppm
Sep 24 07:18:23 dist-test-slave-rc1w krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234303, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:23.053329 4171 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:18:23.053923 4171 webserver.cc:460] Webserver started at http://127.3.246.65:38877/ using document root <none> and password file <none>
I20260924 07:18:23.054171 4171 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:23.054240 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:23.054402 4171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:23.055481 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "28abb436887a464282433660c987a802"
format_stamp: "Formatted at 2026-09-24 07:18:23 on dist-test-slave-rc1w"
server_key: "ab9cb32131e6a1435f3b1f200a81ec98"
server_key_iv: "ecdc5bcf1b8181b9f9088c2e2267211e"
server_key_version: "encryptionkey@0"
I20260924 07:18:23.055850 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "28abb436887a464282433660c987a802"
format_stamp: "Formatted at 2026-09-24 07:18:23 on dist-test-slave-rc1w"
server_key: "ab9cb32131e6a1435f3b1f200a81ec98"
server_key_iv: "ecdc5bcf1b8181b9f9088c2e2267211e"
server_key_version: "encryptionkey@0"
I20260924 07:18:23.057202 4171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:18:23.058097 4186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:23.058324 4171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:23.058429 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "28abb436887a464282433660c987a802"
format_stamp: "Formatted at 2026-09-24 07:18:23 on dist-test-slave-rc1w"
server_key: "ab9cb32131e6a1435f3b1f200a81ec98"
server_key_iv: "ecdc5bcf1b8181b9f9088c2e2267211e"
server_key_version: "encryptionkey@0"
I20260924 07:18:23.058509 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:23.080812 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:23.083384 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:23.083577 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:23.083838 4171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:23.084187 4171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:23.084245 4171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:23.084281 4171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:23.084311 4171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:23.090580 4171 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:41165
I20260924 07:18:23.090629 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:41165 every 8 connection(s)
I20260924 07:18:23.090953 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:23.092837 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 4171
I20260924 07:18:23.092933 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:23.093103 4057 external_mini_cluster.cc:1428] Setting key 81b6990b1bcc8b697511350a20abc6b2
Sep 24 07:18:23 dist-test-slave-rc1w krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234303, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
I20260924 07:18:23.102097 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:23.091758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:41831 (local address 127.3.246.126:34265)
0924 07:18:23.091844 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:23.091847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:23.092067 (+ 220us) server_negotiation.cc:407] Connection header received
0924 07:18:23.092585 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:23.092586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:23.092602 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:23.092707 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:23.094032 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.094807 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.095899 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.096243 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.096317 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:23.098930 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:23.098932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:23.098933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:23.098942 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:23.100620 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:23.101022 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:23.101024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:23.101025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:23.101086 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:23.101328 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:23.101330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:23.101331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:23.101473 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:23.101542 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:23.101880 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 07:18:23.101995 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:18:23.102141 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:23.091710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:34265 (local address 127.3.246.65:41831)
0924 07:18:23.091901 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:23.091939 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:23.092428 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:23.092798 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:23.092802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:23.093609 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:23.093931 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:23.093937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.094902 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.094903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:23.095801 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:23.095803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.096367 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.096368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:23.096494 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:23.097158 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:23.097172 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:23.098821 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:23.100704 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:23.100710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:23.100712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:23.100938 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:23.101160 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:23.101160 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:23.101161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:23.101255 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:23.101542 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:23.101547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:23.101666 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 07:18:23.101840 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 07:18:23.101969 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20260924 07:18:23.102735 4300 heartbeater.cc:347] Connected to a master server at 127.3.246.126:34265
I20260924 07:18:23.102883 4300 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:23.103091 4300 heartbeater.cc:511] Master 127.3.246.126:34265 requested a full tablet report, sending...
I20260924 07:18:23.103667 4109 ts_manager.cc:194] Registered new tserver with Master: 28abb436887a464282433660c987a802 (127.3.246.65:41165)
I20260924 07:18:23.104398 4109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:41831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:18:23.121229 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34265
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:23.204370 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:23.204586 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:23.204609 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:23.205791 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:23.205834 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:23.205862 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:23.207309 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34265
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:23.207573 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:23.207810 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:18:23.213140 4305 server_base.cc:1034] running on GCE node
W20260924 07:18:23.213037 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:23.213037 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:23.213043 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:23.213485 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:23.213750 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:23.214921 4305 hybrid_clock.cc:648] HybridClock initialized: now 1790234303214888 us; error 34 us; skew 500 ppm
Sep 24 07:18:23 dist-test-slave-rc1w krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234303, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:23.217401 4305 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:18:23.217952 4305 webserver.cc:460] Webserver started at http://127.3.246.66:38459/ using document root <none> and password file <none>
I20260924 07:18:23.218178 4305 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:23.218245 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:23.218396 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:23.219471 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "c07d87758cd842d7ae18099a1c497ed7"
format_stamp: "Formatted at 2026-09-24 07:18:23 on dist-test-slave-rc1w"
server_key: "d652c94ab9cea84037a5358ee8bbabb5"
server_key_iv: "45e514d78627a4f06c1a98f904ce9225"
server_key_version: "encryptionkey@0"
I20260924 07:18:23.219841 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "c07d87758cd842d7ae18099a1c497ed7"
format_stamp: "Formatted at 2026-09-24 07:18:23 on dist-test-slave-rc1w"
server_key: "d652c94ab9cea84037a5358ee8bbabb5"
server_key_iv: "45e514d78627a4f06c1a98f904ce9225"
server_key_version: "encryptionkey@0"
I20260924 07:18:23.221155 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:18:23.221916 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:23.222085 4305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:23.222185 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "c07d87758cd842d7ae18099a1c497ed7"
format_stamp: "Formatted at 2026-09-24 07:18:23 on dist-test-slave-rc1w"
server_key: "d652c94ab9cea84037a5358ee8bbabb5"
server_key_iv: "45e514d78627a4f06c1a98f904ce9225"
server_key_version: "encryptionkey@0"
I20260924 07:18:23.222265 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:23.231247 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:23.233187 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:23.233354 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:23.233603 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:23.233949 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:23.234007 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:23.234040 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:23.234069 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:23.240260 4305 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:41821
I20260924 07:18:23.240303 4433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:41821 every 8 connection(s)
I20260924 07:18:23.240716 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:23.246152 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 4305
I20260924 07:18:23.246255 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:23.246419 4057 external_mini_cluster.cc:1428] Setting key fc78e36093e4826a1d8f1fa4c291819f
Sep 24 07:18:23 dist-test-slave-rc1w krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234303, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:23.250774 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:23.241427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:34265 (local address 127.3.246.66:37511)
0924 07:18:23.241601 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:23.241643 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:23.242167 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:23.242439 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:23.242442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:23.243003 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:23.243200 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:23.243205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.243944 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.243945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:23.244776 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:23.244779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.245193 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.245194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:23.245295 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:23.245866 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:23.245874 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:23.247575 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:23.249305 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:23.249312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:23.249315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:23.249638 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:23.249891 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:23.249891 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:23.249892 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:23.249997 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:23.250298 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:23.250302 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:23.250419 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 07:18:23.250495 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:18:23.250612 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260924 07:18:23.250811 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:23.241479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:37511 (local address 127.3.246.126:34265)
0924 07:18:23.241524 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:23.241527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:23.241761 (+ 234us) server_negotiation.cc:407] Connection header received
0924 07:18:23.242280 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:23.242282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:23.242295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:23.242339 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:23.243304 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.243824 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.244874 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.245106 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.245175 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:23.247667 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:23.247669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:23.247669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:23.247678 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:23.249216 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:23.249742 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:23.249743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:23.249744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:23.249817 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:23.250077 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:23.250078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:23.250078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:23.250233 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:23.250308 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:23.250567 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 07:18:23.250662 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:18:23.251498 4434 heartbeater.cc:347] Connected to a master server at 127.3.246.126:34265
I20260924 07:18:23.251628 4434 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:23.251843 4434 heartbeater.cc:511] Master 127.3.246.126:34265 requested a full tablet report, sending...
I20260924 07:18:23.252280 4109 ts_manager.cc:194] Registered new tserver with Master: c07d87758cd842d7ae18099a1c497ed7 (127.3.246.66:41821)
I20260924 07:18:23.252677 4109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:37511
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:18:23.273229 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34265
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:23.359508 4439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:23.359730 4439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:23.359755 4439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:23.361011 4439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:23.361052 4439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:23.361078 4439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:23.362546 4439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34265
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:23.362741 4439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:23.362932 4439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:23.368657 4444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:23.368619 4447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:23.368619 4445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:23.369063 4439 server_base.cc:1034] running on GCE node
I20260924 07:18:23.369253 4439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:23.369460 4439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:23.370604 4439 hybrid_clock.cc:648] HybridClock initialized: now 1790234303370562 us; error 57 us; skew 500 ppm
Sep 24 07:18:23 dist-test-slave-rc1w krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234303, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:23.373361 4439 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:18:23.373960 4439 webserver.cc:460] Webserver started at http://127.3.246.67:43895/ using document root <none> and password file <none>
I20260924 07:18:23.374195 4439 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:23.374265 4439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:23.374423 4439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:23.375525 4439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "43ee4a8e1bd24852b0b474e7323cd899"
format_stamp: "Formatted at 2026-09-24 07:18:23 on dist-test-slave-rc1w"
server_key: "35aef2565cc60846d06ac14e2031e7ed"
server_key_iv: "fcceb82f28cd34d4de72992841be9ada"
server_key_version: "encryptionkey@0"
I20260924 07:18:23.375905 4439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "43ee4a8e1bd24852b0b474e7323cd899"
format_stamp: "Formatted at 2026-09-24 07:18:23 on dist-test-slave-rc1w"
server_key: "35aef2565cc60846d06ac14e2031e7ed"
server_key_iv: "fcceb82f28cd34d4de72992841be9ada"
server_key_version: "encryptionkey@0"
I20260924 07:18:23.377290 4439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:18:23.378249 4454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:23.378564 4439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:18:23.378703 4439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "43ee4a8e1bd24852b0b474e7323cd899"
format_stamp: "Formatted at 2026-09-24 07:18:23 on dist-test-slave-rc1w"
server_key: "35aef2565cc60846d06ac14e2031e7ed"
server_key_iv: "fcceb82f28cd34d4de72992841be9ada"
server_key_version: "encryptionkey@0"
I20260924 07:18:23.378783 4439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:23.395967 4439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:23.398286 4439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:23.398463 4439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:23.398718 4439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:23.399063 4439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:23.399120 4439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:23.399158 4439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:23.399199 4439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:23.405359 4439 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:39299
I20260924 07:18:23.405445 4567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:39299 every 8 connection(s)
I20260924 07:18:23.405833 4439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:23.408699 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 4439
I20260924 07:18:23.408808 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:23.409086 4057 external_mini_cluster.cc:1428] Setting key 1f84d87c76ec226cfa40eb640a1bcdc7
Sep 24 07:18:23 dist-test-slave-rc1w krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234303, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:23.416635 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:23.406675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:35359 (local address 127.3.246.126:34265)
0924 07:18:23.406719 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:23.406721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:23.406972 (+ 251us) server_negotiation.cc:407] Connection header received
0924 07:18:23.407624 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:23.407625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:23.407645 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:23.407735 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:23.409009 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.409811 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.411053 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.411253 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.411317 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:23.413725 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:23.413727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:23.413728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:23.413736 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:23.415279 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:23.415703 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:23.415703 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:23.415704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:23.415757 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:23.416013 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:23.416014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:23.416015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:23.416139 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:23.416215 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:23.416498 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 07:18:23.416567 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:18:23.416812 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:23.406646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:34265 (local address 127.3.246.67:35359)
0924 07:18:23.406835 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:23.406882 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:23.407478 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:23.407831 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:23.407834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:23.408541 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:23.408854 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:23.408860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.409952 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.409954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:23.410955 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:23.410959 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.411343 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.411344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:23.411448 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:23.412061 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:23.412069 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:23.413563 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:23.415387 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:23.415393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:23.415395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:23.415627 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:23.415836 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:23.415836 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:23.415837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:23.415934 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:23.416234 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:23.416239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:23.416360 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 07:18:23.416450 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:18:23.416573 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260924 07:18:23.417323 4568 heartbeater.cc:347] Connected to a master server at 127.3.246.126:34265
I20260924 07:18:23.417419 4568 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:23.417568 4568 heartbeater.cc:511] Master 127.3.246.126:34265 requested a full tablet report, sending...
I20260924 07:18:23.417990 4109 ts_manager.cc:194] Registered new tserver with Master: 43ee4a8e1bd24852b0b474e7323cd899 (127.3.246.67:39299)
I20260924 07:18:23.418359 4109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:35359
I20260924 07:18:23.421247 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:18:23 dist-test-slave-rc1w krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:18:23 dist-test-slave-rc1w krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.246.65@KRBTEST.COM
I20260924 07:18:23.445700 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:23.436849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.3.246.65:41165)
0924 07:18:23.437048 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:23.437053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:23.437064 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:18:23.437115 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:23.437117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:23.437131 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:23.437197 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:23.438032 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.438950 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.439700 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.440039 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.440098 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:23.442733 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:23.442744 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:23.442745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:23.442757 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:23.444410 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:23.444787 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:23.444790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:23.444791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:23.444862 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:23.445071 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:23.445073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:23.445073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:23.445221 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:23.445370 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:23.445460 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 07:18:23.445543 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":114,"threads_started":1}
W20260924 07:18:23.446099 4214 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44470
Sep 24 07:18:23 dist-test-slave-rc1w krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:18:23 dist-test-slave-rc1w krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.65@KRBTEST.COM
I20260924 07:18:23.469061 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:23.461771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44486 (local address 127.3.246.65:41165)
0924 07:18:23.461833 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:23.461835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:23.461845 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:18:23.461925 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:23.461926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:23.461943 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:23.462014 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:23.462590 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.463089 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.463852 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:23.464059 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:23.464145 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:23.466484 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:23.466485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:23.466486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:23.466493 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:23.467822 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:23.468260 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:23.468261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:23.468262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:23.468324 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:23.468538 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:23.468539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:23.468539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:23.468647 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:23.468775 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:23.468865 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 07:18:23.468951 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:18:23.469656 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 4171
I20260924 07:18:23.474635 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 4305
I20260924 07:18:23.478850 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 4439
I20260924 07:18:23.483172 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 4077
2026-09-24T07:18:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1872 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:23 dist-test-slave-rc1w krb5kdc[4582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:23 dist-test-slave-rc1w krb5kdc[4582](info): set up 2 sockets
Sep 24 07:18:23 dist-test-slave-rc1w krb5kdc[4582](info): commencing operation
krb5kdc: starting...
W20260924 07:18:24.529388 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:24 dist-test-slave-rc1w krb5kdc[4582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:18:24.648934 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37811
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:37811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:24.733824 4595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:24.734037 4595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:24.734061 4595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:24.735312 4595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:24.735357 4595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:24.735373 4595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:24.735450 4595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:24.735481 4595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:24.736987 4595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:37811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:24.737237 4595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:24.737491 4595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:24.742513 4601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:24.742487 4600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:24.742502 4603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:24.742865 4595 server_base.cc:1034] running on GCE node
I20260924 07:18:24.743135 4595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:24.743451 4595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:24.744619 4595 hybrid_clock.cc:648] HybridClock initialized: now 1790234304744602 us; error 29 us; skew 500 ppm
Sep 24 07:18:24 dist-test-slave-rc1w krb5kdc[4582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234304, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:24.747134 4595 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:18:24.747696 4595 webserver.cc:460] Webserver started at http://127.3.246.126:40675/ using document root <none> and password file <none>
I20260924 07:18:24.747961 4595 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:24.748034 4595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:24.748191 4595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:24.749264 4595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "7b9e655fb153457b82b9cc3ce35794a7"
format_stamp: "Formatted at 2026-09-24 07:18:24 on dist-test-slave-rc1w"
server_key: "85e1cfe4617da1cec6bf109bbcae9437"
server_key_iv: "883a760fcc53c4747770b118a9316df7"
server_key_version: "encryptionkey@0"
I20260924 07:18:24.749647 4595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "7b9e655fb153457b82b9cc3ce35794a7"
format_stamp: "Formatted at 2026-09-24 07:18:24 on dist-test-slave-rc1w"
server_key: "85e1cfe4617da1cec6bf109bbcae9437"
server_key_iv: "883a760fcc53c4747770b118a9316df7"
server_key_version: "encryptionkey@0"
I20260924 07:18:24.751032 4595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:24.751878 4610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:24.752106 4595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:24.752224 4595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "7b9e655fb153457b82b9cc3ce35794a7"
format_stamp: "Formatted at 2026-09-24 07:18:24 on dist-test-slave-rc1w"
server_key: "85e1cfe4617da1cec6bf109bbcae9437"
server_key_iv: "883a760fcc53c4747770b118a9316df7"
server_key_version: "encryptionkey@0"
I20260924 07:18:24.752329 4595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:24.774029 4595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:24.776114 4595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:24.776281 4595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:24.780164 4595 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:37811
I20260924 07:18:24.780205 4662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:37811 every 8 connection(s)
I20260924 07:18:24.780576 4595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:24.781458 4663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:24.783752 4663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7: Bootstrap starting.
I20260924 07:18:24.784063 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 4595
I20260924 07:18:24.784162 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:24.784332 4057 external_mini_cluster.cc:1428] Setting key afcbe5ce4b578be4ec953ab19684be1d
I20260924 07:18:24.784420 4663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:24.784750 4663 log.cc:824] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:24.785467 4663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7: No bootstrap required, opened a new log
I20260924 07:18:24.787362 4663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9e655fb153457b82b9cc3ce35794a7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37811 } }
I20260924 07:18:24.787516 4663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:24.787562 4663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9e655fb153457b82b9cc3ce35794a7, State: Initialized, Role: FOLLOWER
I20260924 07:18:24.787693 4663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9e655fb153457b82b9cc3ce35794a7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37811 } }
I20260924 07:18:24.787789 4663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:24.787832 4663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:24.787880 4663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:24.788393 4663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9e655fb153457b82b9cc3ce35794a7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37811 } }
I20260924 07:18:24.788512 4663 leader_election.cc:302] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b9e655fb153457b82b9cc3ce35794a7; no voters:
I20260924 07:18:24.788697 4663 leader_election.cc:288] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:24.788735 4668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:24.788882 4668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [term 1 LEADER]: Becoming Leader. State: Replica: 7b9e655fb153457b82b9cc3ce35794a7, State: Running, Role: LEADER
I20260924 07:18:24.789031 4663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:24.789012 4668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9e655fb153457b82b9cc3ce35794a7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37811 } }
W20260924 07:18:24.791064 4683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:18:24.791122 4683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 07:18:24 dist-test-slave-rc1w krb5kdc[4582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:24.791669 4669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b9e655fb153457b82b9cc3ce35794a7. Latest consensus state: current_term: 1 leader_uuid: "7b9e655fb153457b82b9cc3ce35794a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9e655fb153457b82b9cc3ce35794a7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37811 } } }
I20260924 07:18:24.791756 4669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:24.791950 4670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b9e655fb153457b82b9cc3ce35794a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9e655fb153457b82b9cc3ce35794a7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37811 } } }
I20260924 07:18:24.792033 4670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:24.792138 4685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:24.792332 4685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:24.793696 4685 catalog_manager.cc:1347] Generated new cluster ID: 9c6a6a9124dd4fa1a89079a73ede27c9
I20260924 07:18:24.793742 4685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:24.796077 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:24.784919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.3.246.126:37811)
0924 07:18:24.785103 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:24.785111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:24.785147 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:18:24.785582 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:24.785591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:24.785650 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:24.785834 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:24.786436 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:24.787360 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:24.788281 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:24.788854 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:24.789326 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:24.791919 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:24.791923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:24.791923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:24.791935 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:24.794075 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:24.794488 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:24.794491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:24.794492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:24.794593 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:24.794799 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:24.794801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:24.794801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:24.795105 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:24.795258 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:24.795726 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 07:18:24.795860 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260924 07:18:24.829345 4685 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:24.829973 4685 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:24.834856 4685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b9e655fb153457b82b9cc3ce35794a7: Generated new TSK 0
I20260924 07:18:24.835057 4685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:18:24.869356 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37811
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:24.951061 4689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:24.951278 4689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:24.951303 4689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:24.952399 4689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:24.952435 4689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:24.952461 4689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:24.953989 4689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:24.954185 4689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:24.954365 4689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:24.959589 4697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:24.959738 4694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:24.959661 4689 server_base.cc:1034] running on GCE node
W20260924 07:18:24.959601 4695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:24.960038 4689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:24.960294 4689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:24.961460 4689 hybrid_clock.cc:648] HybridClock initialized: now 1790234304961432 us; error 62 us; skew 500 ppm
Sep 24 07:18:24 dist-test-slave-rc1w krb5kdc[4582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234304, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:24.964097 4689 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:18:24.964591 4689 webserver.cc:460] Webserver started at http://127.3.246.65:45691/ using document root <none> and password file <none>
I20260924 07:18:24.964854 4689 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:24.964926 4689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:24.965073 4689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:24.966178 4689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "da11eb632b91440dad25489c4d3b7d9d"
format_stamp: "Formatted at 2026-09-24 07:18:24 on dist-test-slave-rc1w"
server_key: "c539dc2832c88a2bb0581f610daf648f"
server_key_iv: "17bc9711d06db45858746455c4f95624"
server_key_version: "encryptionkey@0"
I20260924 07:18:24.966527 4689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "da11eb632b91440dad25489c4d3b7d9d"
format_stamp: "Formatted at 2026-09-24 07:18:24 on dist-test-slave-rc1w"
server_key: "c539dc2832c88a2bb0581f610daf648f"
server_key_iv: "17bc9711d06db45858746455c4f95624"
server_key_version: "encryptionkey@0"
I20260924 07:18:24.967859 4689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 07:18:24.968585 4704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:24.968770 4689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:24.968868 4689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "da11eb632b91440dad25489c4d3b7d9d"
format_stamp: "Formatted at 2026-09-24 07:18:24 on dist-test-slave-rc1w"
server_key: "c539dc2832c88a2bb0581f610daf648f"
server_key_iv: "17bc9711d06db45858746455c4f95624"
server_key_version: "encryptionkey@0"
I20260924 07:18:24.968948 4689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:24.979841 4689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:24.982185 4689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:24.982352 4689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:24.982573 4689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:24.982884 4689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:24.982939 4689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:24.982980 4689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:24.983007 4689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:24.989267 4689 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:42571
I20260924 07:18:24.989368 4817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:42571 every 8 connection(s)
I20260924 07:18:24.989713 4689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:24.994194 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 4689
I20260924 07:18:24.994272 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:24.994445 4057 external_mini_cluster.cc:1428] Setting key ef13f60218e2a0019a72354b27854ea5
Sep 24 07:18:24 dist-test-slave-rc1w krb5kdc[4582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234304, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:25.000305 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:24.990678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:57079 (local address 127.3.246.126:37811)
0924 07:18:24.990787 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:24.990789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:24.990991 (+ 202us) server_negotiation.cc:407] Connection header received
0924 07:18:24.991960 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:24.991962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:24.991973 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:24.992022 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:24.992663 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:24.993173 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:24.994386 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:24.994669 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:24.994758 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:24.997106 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:24.997108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:24.997108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:24.997116 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:24.998878 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:24.999322 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:24.999323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:24.999324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:24.999385 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:24.999642 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:24.999643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:24.999643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:24.999764 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:24.999853 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:25.000125 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 07:18:25.000211 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 07:18:25.000578 4820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:24.990669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37811 (local address 127.3.246.65:57079)
0924 07:18:24.990861 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:24.990908 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:24.991848 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:24.992128 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:24.992132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:24.992383 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:24.992564 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:24.992568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:24.993277 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:24.993278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:24.994233 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:24.994236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:24.994766 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:24.994767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:24.994928 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:24.995535 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:24.995543 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:24.997010 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:24.998977 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:24.998984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:24.998986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:24.999215 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:24.999464 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:24.999464 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:24.999465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:24.999573 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:24.999845 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:24.999852 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:24.999992 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 07:18:25.000104 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 07:18:25.000261 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":90,"threads_started":1}
I20260924 07:18:25.001323 4818 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37811
I20260924 07:18:25.001430 4818 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:25.001600 4818 heartbeater.cc:511] Master 127.3.246.126:37811 requested a full tablet report, sending...
I20260924 07:18:25.002255 4627 ts_manager.cc:194] Registered new tserver with Master: da11eb632b91440dad25489c4d3b7d9d (127.3.246.65:42571)
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
I20260924 07:18:25.003177 4627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:57079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:18:25.020232 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37811
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:25.099884 4823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:25.100096 4823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:25.100121 4823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:25.101210 4823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:25.101244 4823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:25.101270 4823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:25.102689 4823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:25.102878 4823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:25.103044 4823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:25.108448 4828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:25.108443 4831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:25.108558 4823 server_base.cc:1034] running on GCE node
W20260924 07:18:25.108686 4829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:25.108932 4823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:25.109185 4823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:25.110323 4823 hybrid_clock.cc:648] HybridClock initialized: now 1790234305110275 us; error 71 us; skew 500 ppm
Sep 24 07:18:25 dist-test-slave-rc1w krb5kdc[4582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:25.112825 4823 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:18:25.113299 4823 webserver.cc:460] Webserver started at http://127.3.246.66:42119/ using document root <none> and password file <none>
I20260924 07:18:25.113525 4823 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:25.113593 4823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:25.113773 4823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:25.114874 4823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "7512ca61cbc247c3bcbbd01aa036aa50"
format_stamp: "Formatted at 2026-09-24 07:18:25 on dist-test-slave-rc1w"
server_key: "22044a61b0692b73227bc1267c273c12"
server_key_iv: "aeef776a79c2c30b7a5dd5b42f73a4c8"
server_key_version: "encryptionkey@0"
I20260924 07:18:25.115232 4823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "7512ca61cbc247c3bcbbd01aa036aa50"
format_stamp: "Formatted at 2026-09-24 07:18:25 on dist-test-slave-rc1w"
server_key: "22044a61b0692b73227bc1267c273c12"
server_key_iv: "aeef776a79c2c30b7a5dd5b42f73a4c8"
server_key_version: "encryptionkey@0"
I20260924 07:18:25.116567 4823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:18:25.117362 4838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:25.117558 4823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:25.117655 4823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "7512ca61cbc247c3bcbbd01aa036aa50"
format_stamp: "Formatted at 2026-09-24 07:18:25 on dist-test-slave-rc1w"
server_key: "22044a61b0692b73227bc1267c273c12"
server_key_iv: "aeef776a79c2c30b7a5dd5b42f73a4c8"
server_key_version: "encryptionkey@0"
I20260924 07:18:25.117749 4823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:25.156785 4823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:25.159358 4823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:25.159533 4823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:25.159794 4823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:25.160130 4823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:25.160187 4823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:25.160230 4823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:25.160270 4823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:25.166189 4823 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:46447
I20260924 07:18:25.166232 4951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:46447 every 8 connection(s)
I20260924 07:18:25.166550 4823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:18:25 dist-test-slave-rc1w krb5kdc[4582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:25.175510 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 4823
I20260924 07:18:25.175626 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:25.175796 4057 external_mini_cluster.cc:1428] Setting key 082e604b9a4301590851eb0c560d1638
I20260924 07:18:25.176362 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.167352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:53613 (local address 127.3.246.126:37811)
0924 07:18:25.167423 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:25.167425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:25.167573 (+ 148us) server_negotiation.cc:407] Connection header received
0924 07:18:25.168527 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:25.168528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:25.168538 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:25.168604 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:25.169357 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.169908 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.170912 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.171102 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.171167 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.173564 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:25.173565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:25.173566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:25.173572 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:25.174935 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:25.175344 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:25.175345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:25.175346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:25.175419 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:25.175671 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:25.175672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:25.175672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:25.175800 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:25.175878 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:25.176175 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 07:18:25.176269 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:18:25.176503 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.167274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37811 (local address 127.3.246.66:53613)
0924 07:18:25.167435 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:25.167476 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:25.168414 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:25.168732 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:25.168736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:25.169007 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:25.169236 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.169241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.169993 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.169994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.170814 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.170816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.171182 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.171183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.171285 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.171917 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:25.171926 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:25.173454 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:25.175011 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:25.175017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:25.175019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:25.175234 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:25.175488 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:25.175489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:25.175489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:25.175583 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:25.175869 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:25.175875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:25.176006 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 07:18:25.176133 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 07:18:25.176270 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260924 07:18:25.176977 4952 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37811
I20260924 07:18:25.177067 4952 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:25.177258 4952 heartbeater.cc:511] Master 127.3.246.126:37811 requested a full tablet report, sending...
I20260924 07:18:25.177634 4627 ts_manager.cc:194] Registered new tserver with Master: 7512ca61cbc247c3bcbbd01aa036aa50 (127.3.246.66:46447)
I20260924 07:18:25.178195 4627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:53613
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:18:25.204955 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37811
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:25.285452 4957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:25.285683 4957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:25.285760 4957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:25.286911 4957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:25.286984 4957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:25.287030 4957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:25.288496 4957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:25.288757 4957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:25.289014 4957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:25.294538 4962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:25.294548 4963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:25.294584 4957 server_base.cc:1034] running on GCE node
W20260924 07:18:25.294548 4965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:25.294951 4957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:25.295207 4957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:25.296377 4957 hybrid_clock.cc:648] HybridClock initialized: now 1790234305296374 us; error 35 us; skew 500 ppm
Sep 24 07:18:25 dist-test-slave-rc1w krb5kdc[4582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:25.299132 4957 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:18:25.299602 4957 webserver.cc:460] Webserver started at http://127.3.246.67:41523/ using document root <none> and password file <none>
I20260924 07:18:25.299798 4957 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:25.299841 4957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:25.299937 4957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:25.300967 4957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "6c87341651864c909c2e68942f0e8563"
format_stamp: "Formatted at 2026-09-24 07:18:25 on dist-test-slave-rc1w"
server_key: "4c789f00d287396311135cf2e15a44ec"
server_key_iv: "8df9f3ce2594ae64a5f16a084963f523"
server_key_version: "encryptionkey@0"
I20260924 07:18:25.301282 4957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "6c87341651864c909c2e68942f0e8563"
format_stamp: "Formatted at 2026-09-24 07:18:25 on dist-test-slave-rc1w"
server_key: "4c789f00d287396311135cf2e15a44ec"
server_key_iv: "8df9f3ce2594ae64a5f16a084963f523"
server_key_version: "encryptionkey@0"
I20260924 07:18:25.302718 4957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:25.303594 4972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:25.303797 4957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:25.303890 4957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "6c87341651864c909c2e68942f0e8563"
format_stamp: "Formatted at 2026-09-24 07:18:25 on dist-test-slave-rc1w"
server_key: "4c789f00d287396311135cf2e15a44ec"
server_key_iv: "8df9f3ce2594ae64a5f16a084963f523"
server_key_version: "encryptionkey@0"
I20260924 07:18:25.303963 4957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:25.318990 4957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:25.321524 4957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:25.321650 4957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:25.321907 4957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:25.322160 4957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:25.322188 4957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:25.322213 4957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:25.322233 4957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:25.327831 4957 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:43087
I20260924 07:18:25.327930 5085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:43087 every 8 connection(s)
I20260924 07:18:25.328225 4957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:25.329507 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 4957
I20260924 07:18:25.329603 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:25.329803 4057 external_mini_cluster.cc:1428] Setting key 6652b52af8ad13493b3976d8cb706ec6
Sep 24 07:18:25 dist-test-slave-rc1w krb5kdc[4582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:25.337903 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.329001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:42425 (local address 127.3.246.126:37811)
0924 07:18:25.329075 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:25.329077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:25.329295 (+ 218us) server_negotiation.cc:407] Connection header received
0924 07:18:25.330569 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:25.330570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:25.330582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:25.330634 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:25.331260 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.331770 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.332775 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.332979 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.333062 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.335342 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:25.335343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:25.335343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:25.335355 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:25.336513 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:25.336924 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:25.336926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:25.336926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:25.336989 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:25.337247 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:25.337248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:25.337248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:25.337367 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:25.337443 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:25.337737 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 07:18:25.337815 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:18:25.338078 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.328977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37811 (local address 127.3.246.67:42425)
0924 07:18:25.329162 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:25.329197 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:25.330459 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:25.330717 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:25.330721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:25.330981 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:25.331160 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.331164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.331864 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.331865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.332665 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.332667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.333072 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.333073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.333209 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.333742 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:25.333750 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:25.335201 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:25.336611 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:25.336617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:25.336620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:25.336845 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:25.337056 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:25.337056 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:25.337057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:25.337172 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:25.337433 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:25.337439 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:25.337576 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 07:18:25.337680 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 07:18:25.337848 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260924 07:18:25.338574 5086 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37811
I20260924 07:18:25.338668 5086 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:25.338822 5086 heartbeater.cc:511] Master 127.3.246.126:37811 requested a full tablet report, sending...
I20260924 07:18:25.339205 4627 ts_manager.cc:194] Registered new tserver with Master: 6c87341651864c909c2e68942f0e8563 (127.3.246.67:43087)
I20260924 07:18:25.339532 4627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:42425
I20260924 07:18:25.341389 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:18:25.349167 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.343051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.3.246.126:37811)
0924 07:18:25.343124 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:25.343126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:25.343204 (+ 78us) server_negotiation.cc:407] Connection header received
0924 07:18:25.343270 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:25.343270 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:25.343281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:25.343355 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:25.343900 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.344367 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.345093 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.345286 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.345367 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.346722 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:25.346723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:25.346724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:25.346733 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:25.347973 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:25.348377 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:25.348378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:25.348378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:25.348463 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:25.348688 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:25.348689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:25.348689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:25.348835 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:25.348921 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:25.349013 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 07:18:25.349074 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:18:25.351438 4627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:25.353046 4627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:25.359782 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.355874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.3.246.67:43087)
0924 07:18:25.356169 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:25.356174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:25.356183 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:25.356246 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:25.356248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:25.356262 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:25.356345 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:25.356580 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.357306 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.358799 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.359360 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.359456 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.359517 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:25.359661 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 07:18:25.359709 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":83,"threads_started":1}
I20260924 07:18:25.359774 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.355646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:43087 (local address 127.0.0.1:35878)
0924 07:18:25.355914 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:25.355957 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:25.356150 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:25.356365 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:25.356368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:25.356370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:25.356435 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.356437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.357411 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.357412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.358708 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.358710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.359457 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.359458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.359532 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.359545 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 07:18:25.359641 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 07:18:25.359696 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
I20260924 07:18:25.360134 5102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.355900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36148 (local address 127.3.246.65:42571)
0924 07:18:25.356088 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:25.356093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:25.356233 (+ 140us) server_negotiation.cc:407] Connection header received
0924 07:18:25.356273 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:25.356275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:25.356292 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:25.356359 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:25.356627 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.357712 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.358955 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.359627 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.359709 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.359757 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:25.360015 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 07:18:25.360063 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260924 07:18:25.360165 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.355642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:46447 (local address 127.0.0.1:56744)
0924 07:18:25.355925 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:25.355961 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:25.356023 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:25.356412 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:25.356412 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:25.356414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:25.356444 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.356445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.357818 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.357819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.358710 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.358712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.359797 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.359798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.359867 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.359870 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:25.359922 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 07:18:25.359969 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":17,"thread_start_us":63,"threads_started":1}
I20260924 07:18:25.360194 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.355743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.3.246.66:46447)
0924 07:18:25.356083 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:25.356089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:25.356101 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:18:25.356143 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:25.356145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:25.356161 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:25.356236 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:25.356524 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.357650 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.358801 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.359627 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.359701 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.359757 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:25.360055 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 07:18:25.360099 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":81,"threads_started":1}
I20260924 07:18:25.360092 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.355646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:42571 (local address 127.0.0.1:36148)
0924 07:18:25.356126 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:25.356135 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:25.356178 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:25.356448 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:25.356449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:25.356451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:25.356505 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.356507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.357795 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.357796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.358875 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.358877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.359796 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.359797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.359868 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.359872 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:25.359922 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 07:18:25.359969 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":38,"thread_start_us":78,"threads_started":1}
I20260924 07:18:25.360880 4886 tablet_service.cc:1467] Processing CreateTablet for tablet f25181d8b710409b98f8d6953fe1090e (DEFAULT_TABLE table=test-table [id=9b4e03f4678c45c29f23354707f72036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:25.360879 5020 tablet_service.cc:1467] Processing CreateTablet for tablet f25181d8b710409b98f8d6953fe1090e (DEFAULT_TABLE table=test-table [id=9b4e03f4678c45c29f23354707f72036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:25.361238 5020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25181d8b710409b98f8d6953fe1090e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:25.361238 4886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25181d8b710409b98f8d6953fe1090e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:25.361331 4752 tablet_service.cc:1467] Processing CreateTablet for tablet f25181d8b710409b98f8d6953fe1090e (DEFAULT_TABLE table=test-table [id=9b4e03f4678c45c29f23354707f72036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:25.361649 4752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25181d8b710409b98f8d6953fe1090e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:25.364071 5103 tablet_bootstrap.cc:492] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d: Bootstrap starting.
I20260924 07:18:25.364625 5104 tablet_bootstrap.cc:492] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50: Bootstrap starting.
I20260924 07:18:25.364884 5103 tablet_bootstrap.cc:654] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:25.365309 5104 tablet_bootstrap.cc:654] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:25.365512 5103 log.cc:824] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:25.365823 5105 tablet_bootstrap.cc:492] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563: Bootstrap starting.
I20260924 07:18:25.365892 5104 log.cc:824] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:25.366412 5103 tablet_bootstrap.cc:492] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d: No bootstrap required, opened a new log
I20260924 07:18:25.366483 5103 ts_tablet_manager.cc:1397] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:18:25.366523 5105 tablet_bootstrap.cc:654] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:25.366659 5104 tablet_bootstrap.cc:492] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50: No bootstrap required, opened a new log
I20260924 07:18:25.366724 5104 ts_tablet_manager.cc:1397] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:18:25.367051 5105 log.cc:824] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:25.367901 5105 tablet_bootstrap.cc:492] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563: No bootstrap required, opened a new log
I20260924 07:18:25.367985 5105 ts_tablet_manager.cc:1397] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:18:25.368495 5103 raft_consensus.cc:348] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46447 } } peers { permanent_uuid: "6c87341651864c909c2e68942f0e8563" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43087 } } peers { permanent_uuid: "da11eb632b91440dad25489c4d3b7d9d" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42571 } }
I20260924 07:18:25.368681 5103 raft_consensus.cc:374] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:25.368741 5103 raft_consensus.cc:729] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da11eb632b91440dad25489c4d3b7d9d, State: Initialized, Role: FOLLOWER
I20260924 07:18:25.368692 5104 raft_consensus.cc:348] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46447 } } peers { permanent_uuid: "6c87341651864c909c2e68942f0e8563" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43087 } } peers { permanent_uuid: "da11eb632b91440dad25489c4d3b7d9d" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42571 } }
I20260924 07:18:25.368840 5104 raft_consensus.cc:374] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:25.368877 5104 raft_consensus.cc:729] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ca61cbc247c3bcbbd01aa036aa50, State: Initialized, Role: FOLLOWER
I20260924 07:18:25.368856 5103 consensus_queue.cc:260] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46447 } } peers { permanent_uuid: "6c87341651864c909c2e68942f0e8563" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43087 } } peers { permanent_uuid: "da11eb632b91440dad25489c4d3b7d9d" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42571 } }
I20260924 07:18:25.369004 5104 consensus_queue.cc:260] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46447 } } peers { permanent_uuid: "6c87341651864c909c2e68942f0e8563" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43087 } } peers { permanent_uuid: "da11eb632b91440dad25489c4d3b7d9d" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42571 } }
I20260924 07:18:25.369182 4818 heartbeater.cc:503] Master 127.3.246.126:37811 was elected leader, sending a full tablet report...
I20260924 07:18:25.369278 5104 ts_tablet_manager.cc:1428] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:25.369433 4952 heartbeater.cc:503] Master 127.3.246.126:37811 was elected leader, sending a full tablet report...
I20260924 07:18:25.369618 5103 ts_tablet_manager.cc:1428] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:25.369948 5105 raft_consensus.cc:348] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46447 } } peers { permanent_uuid: "6c87341651864c909c2e68942f0e8563" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43087 } } peers { permanent_uuid: "da11eb632b91440dad25489c4d3b7d9d" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42571 } }
I20260924 07:18:25.370117 5105 raft_consensus.cc:374] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:25.370162 5105 raft_consensus.cc:729] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c87341651864c909c2e68942f0e8563, State: Initialized, Role: FOLLOWER
I20260924 07:18:25.370288 5105 consensus_queue.cc:260] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46447 } } peers { permanent_uuid: "6c87341651864c909c2e68942f0e8563" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43087 } } peers { permanent_uuid: "da11eb632b91440dad25489c4d3b7d9d" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42571 } }
I20260924 07:18:25.370543 5105 ts_tablet_manager.cc:1428] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:25.370568 5086 heartbeater.cc:503] Master 127.3.246.126:37811 was elected leader, sending a full tablet report...
W20260924 07:18:25.417173 4953 tablet.cc:2369] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:18:25.490510 4819 tablet.cc:2369] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:18:25.538874 5110 raft_consensus.cc:482] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:25.539055 5110 raft_consensus.cc:504] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46447 } } peers { permanent_uuid: "6c87341651864c909c2e68942f0e8563" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43087 } } peers { permanent_uuid: "da11eb632b91440dad25489c4d3b7d9d" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42571 } }
I20260924 07:18:25.539491 5110 leader_election.cc:288] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c87341651864c909c2e68942f0e8563 (127.3.246.67:43087), da11eb632b91440dad25489c4d3b7d9d (127.3.246.65:42571)
I20260924 07:18:25.543659 5102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.539920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:52263 (local address 127.3.246.65:42571)
0924 07:18:25.539972 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:25.539974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:25.540110 (+ 136us) server_negotiation.cc:407] Connection header received
0924 07:18:25.540151 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:25.540152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:25.540163 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:25.540243 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:25.540434 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.541120 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.542666 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.543275 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.543361 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.543371 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:25.543548 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 07:18:25.543588 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:18:25.543659 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.539952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:40899 (local address 127.3.246.67:43087)
0924 07:18:25.540008 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:25.540010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:25.540023 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:18:25.540053 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:25.540054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:25.540064 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:25.540121 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:25.540286 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.541083 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.542665 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.543295 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.543371 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.543380 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:25.543547 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 07:18:25.543588 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 07:18:25.543681 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.539743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:43087 (local address 127.3.246.66:40899)
0924 07:18:25.539809 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:25.539824 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:25.539886 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:25.540131 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:25.540132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:25.540133 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:25.540205 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.540208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.541221 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.541222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.542553 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.542556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.543375 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.543376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.543461 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.543464 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:25.543526 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 07:18:25.543573 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":7}
I20260924 07:18:25.543689 5112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.539828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:42571 (local address 127.3.246.66:52263)
0924 07:18:25.539993 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:25.540003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:25.540057 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:25.540265 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:25.540266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:25.540269 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:25.540347 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.540349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.541221 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.541222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.542553 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:25.542556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.543375 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.543376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:25.543460 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.543465 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:18:25.543526 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:18:25.543573 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":107,"thread_start_us":104,"threads_started":1}
I20260924 07:18:25.544092 5040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25181d8b710409b98f8d6953fe1090e" candidate_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c87341651864c909c2e68942f0e8563" is_pre_election: true
I20260924 07:18:25.544186 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25181d8b710409b98f8d6953fe1090e" candidate_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da11eb632b91440dad25489c4d3b7d9d" is_pre_election: true
I20260924 07:18:25.544317 5040 raft_consensus.cc:2454] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ca61cbc247c3bcbbd01aa036aa50 in term 0.
I20260924 07:18:25.544363 4772 raft_consensus.cc:2454] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ca61cbc247c3bcbbd01aa036aa50 in term 0.
I20260924 07:18:25.544555 4840 leader_election.cc:302] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c87341651864c909c2e68942f0e8563, 7512ca61cbc247c3bcbbd01aa036aa50; no voters:
I20260924 07:18:25.544687 5110 raft_consensus.cc:2781] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:25.544741 5110 raft_consensus.cc:482] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:25.544766 5110 raft_consensus.cc:3037] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:25.545468 5110 raft_consensus.cc:504] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46447 } } peers { permanent_uuid: "6c87341651864c909c2e68942f0e8563" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43087 } } peers { permanent_uuid: "da11eb632b91440dad25489c4d3b7d9d" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42571 } }
I20260924 07:18:25.545620 5110 leader_election.cc:288] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [CANDIDATE]: Term 1 election: Requested vote from peers 6c87341651864c909c2e68942f0e8563 (127.3.246.67:43087), da11eb632b91440dad25489c4d3b7d9d (127.3.246.65:42571)
I20260924 07:18:25.545842 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25181d8b710409b98f8d6953fe1090e" candidate_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da11eb632b91440dad25489c4d3b7d9d"
I20260924 07:18:25.545928 4772 raft_consensus.cc:3037] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:25.546659 4772 raft_consensus.cc:2454] T f25181d8b710409b98f8d6953fe1090e P da11eb632b91440dad25489c4d3b7d9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ca61cbc247c3bcbbd01aa036aa50 in term 1.
I20260924 07:18:25.546788 5040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25181d8b710409b98f8d6953fe1090e" candidate_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c87341651864c909c2e68942f0e8563"
I20260924 07:18:25.546890 4842 leader_election.cc:302] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ca61cbc247c3bcbbd01aa036aa50, da11eb632b91440dad25489c4d3b7d9d; no voters:
I20260924 07:18:25.546882 5040 raft_consensus.cc:3037] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:25.546981 5110 raft_consensus.cc:2781] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:25.547140 5110 raft_consensus.cc:686] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ca61cbc247c3bcbbd01aa036aa50, State: Running, Role: LEADER
I20260924 07:18:25.547237 5110 consensus_queue.cc:237] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46447 } } peers { permanent_uuid: "6c87341651864c909c2e68942f0e8563" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43087 } } peers { permanent_uuid: "da11eb632b91440dad25489c4d3b7d9d" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42571 } }
I20260924 07:18:25.547770 5040 raft_consensus.cc:2454] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ca61cbc247c3bcbbd01aa036aa50 in term 1.
I20260924 07:18:25.548105 4627 catalog_manager.cc:5495] T f25181d8b710409b98f8d6953fe1090e P 7512ca61cbc247c3bcbbd01aa036aa50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ca61cbc247c3bcbbd01aa036aa50 (127.3.246.66). New cstate: current_term: 1 leader_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7512ca61cbc247c3bcbbd01aa036aa50" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c87341651864c909c2e68942f0e8563" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da11eb632b91440dad25489c4d3b7d9d" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42571 } health_report { overall_health: UNKNOWN } } }
W20260924 07:18:25.578862 5087 tablet.cc:2369] T f25181d8b710409b98f8d6953fe1090e P 6c87341651864c909c2e68942f0e8563: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 07:18:25 dist-test-slave-rc1w krb5kdc[4582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.65@KRBTEST.COM
I20260924 07:18:25.595252 5102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.586697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.3.246.65:42571)
0924 07:18:25.586814 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:25.586816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:25.586950 (+ 134us) server_negotiation.cc:407] Connection header received
0924 07:18:25.587041 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:25.587042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:25.587053 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:25.587127 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:25.587716 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.588209 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.589029 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.589266 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.589346 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.592181 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:25.592192 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:25.592192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:25.592199 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:25.594082 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:25.594458 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:25.594461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:25.594461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:25.594533 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:25.594756 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:25.594757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:25.594758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:25.594914 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:25.595039 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:25.595094 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 07:18:25.595160 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 07:18:25 dist-test-slave-rc1w krb5kdc[4582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:18:25 dist-test-slave-rc1w krb5kdc[4582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.246.65@KRBTEST.COM
I20260924 07:18:25.618353 5102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.610583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36170 (local address 127.3.246.65:42571)
0924 07:18:25.610682 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:25.610685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:25.610793 (+ 108us) server_negotiation.cc:407] Connection header received
0924 07:18:25.610892 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:25.610892 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:25.610902 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:25.610965 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:25.611569 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.612061 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.612837 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.613027 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.613105 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.615449 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:25.615451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:25.615451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:25.615458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:25.616881 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:25.617234 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:25.617235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:25.617236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:25.617292 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:25.617542 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:25.617543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:25.617543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:25.617651 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:25.617938 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:25.618141 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 07:18:25.618235 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 07:18:25 dist-test-slave-rc1w krb5kdc[4582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:18:25 dist-test-slave-rc1w krb5kdc[4582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.65@KRBTEST.COM
I20260924 07:18:25.642457 5102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:25.634860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.3.246.65:42571)
0924 07:18:25.634964 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:25.634966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:25.635054 (+ 88us) server_negotiation.cc:407] Connection header received
0924 07:18:25.635135 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:25.635136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:25.635155 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:25.635221 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:25.635808 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.636286 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.637065 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:25.637257 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:25.637338 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:25.639781 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:25.639782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:25.639782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:25.639789 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:25.641162 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:25.641524 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:25.641525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:25.641525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:25.641590 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:25.641845 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:25.641845 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:25.641846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:25.641989 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:25.642109 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:25.642240 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 07:18:25.642324 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 07:18:25.642958 4732 tablet_service.cc:2908] Rejecting scan request for tablet f25181d8b710409b98f8d6953fe1090e: Uninitialized: clean time has not yet been initialized
I20260924 07:18:25.643782 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 4689
I20260924 07:18:25.649389 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 4823
I20260924 07:18:25.654356 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 4957
I20260924 07:18:25.658686 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 4595
2026-09-24T07:18:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2175 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:25 dist-test-slave-rc1w krb5kdc[5127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:25 dist-test-slave-rc1w krb5kdc[5127](info): set up 2 sockets
Sep 24 07:18:25 dist-test-slave-rc1w krb5kdc[5127](info): commencing operation
krb5kdc: starting...
W20260924 07:18:26.725323 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:26 dist-test-slave-rc1w krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:18:26.845667 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:46425
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:46425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:26.925057 5140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:26.925261 5140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:26.925283 5140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:26.926373 5140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:26.926409 5140 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:18:26.926421 5140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:26.926434 5140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:26.926446 5140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:26.926458 5140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:26.927831 5140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:46425
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:46425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:26.928015 5140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:26.928184 5140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:26.932981 5148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:26.933101 5140 server_base.cc:1034] running on GCE node
W20260924 07:18:26.933029 5146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:26.932960 5145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:26.933488 5140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:26.933836 5140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:26.935019 5140 hybrid_clock.cc:648] HybridClock initialized: now 1790234306935008 us; error 32 us; skew 500 ppm
Sep 24 07:18:26 dist-test-slave-rc1w krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:26.937541 5140 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:18:26.938223 5140 webserver.cc:460] Webserver started at http://127.3.246.126:46671/ using document root <none> and password file <none>
I20260924 07:18:26.938436 5140 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:26.938477 5140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:26.938575 5140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:26.939606 5140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "910684c3eb14420490d62ee5c9bee33e"
format_stamp: "Formatted at 2026-09-24 07:18:26 on dist-test-slave-rc1w"
server_key: "576e575ebbfc63a7dbe94367c0b55a0c"
server_key_iv: "540a62d8ec79e5ceade3eefeaa1c9fc9"
server_key_version: "encryptionkey@0"
I20260924 07:18:26.939978 5140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "910684c3eb14420490d62ee5c9bee33e"
format_stamp: "Formatted at 2026-09-24 07:18:26 on dist-test-slave-rc1w"
server_key: "576e575ebbfc63a7dbe94367c0b55a0c"
server_key_iv: "540a62d8ec79e5ceade3eefeaa1c9fc9"
server_key_version: "encryptionkey@0"
I20260924 07:18:26.941293 5140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:26.942094 5155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:26.942293 5140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:18:26.942377 5140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "910684c3eb14420490d62ee5c9bee33e"
format_stamp: "Formatted at 2026-09-24 07:18:26 on dist-test-slave-rc1w"
server_key: "576e575ebbfc63a7dbe94367c0b55a0c"
server_key_iv: "540a62d8ec79e5ceade3eefeaa1c9fc9"
server_key_version: "encryptionkey@0"
I20260924 07:18:26.942433 5140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:26.987916 5140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:26.989953 5140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:26.990072 5140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:26.994596 5140 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:46425
I20260924 07:18:26.994691 5217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:46425 every 8 connection(s)
I20260924 07:18:26.995024 5140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:26.995783 5218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:26.998205 5218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e: Bootstrap starting.
I20260924 07:18:26.998798 5218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:26.999101 5218 log.cc:824] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:26.999786 5218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e: No bootstrap required, opened a new log
I20260924 07:18:27.000789 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 5140
I20260924 07:18:27.000866 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:27.001034 4057 external_mini_cluster.cc:1428] Setting key 7d447d7491d6498df1c3694dea9f7026
I20260924 07:18:27.002437 5218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910684c3eb14420490d62ee5c9bee33e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46425 } }
I20260924 07:18:27.002614 5218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:27.002650 5218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910684c3eb14420490d62ee5c9bee33e, State: Initialized, Role: FOLLOWER
I20260924 07:18:27.002810 5218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910684c3eb14420490d62ee5c9bee33e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46425 } }
I20260924 07:18:27.002878 5218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:27.002900 5218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:27.002970 5218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:27.003576 5218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910684c3eb14420490d62ee5c9bee33e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46425 } }
I20260924 07:18:27.003682 5218 leader_election.cc:302] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 910684c3eb14420490d62ee5c9bee33e; no voters:
I20260924 07:18:27.003831 5218 leader_election.cc:288] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:27.003897 5223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:27.004042 5223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [term 1 LEADER]: Becoming Leader. State: Replica: 910684c3eb14420490d62ee5c9bee33e, State: Running, Role: LEADER
I20260924 07:18:27.004082 5218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:27.004184 5223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910684c3eb14420490d62ee5c9bee33e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46425 } }
I20260924 07:18:27.005499 5225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 910684c3eb14420490d62ee5c9bee33e. Latest consensus state: current_term: 1 leader_uuid: "910684c3eb14420490d62ee5c9bee33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910684c3eb14420490d62ee5c9bee33e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46425 } } }
I20260924 07:18:27.005627 5224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "910684c3eb14420490d62ee5c9bee33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910684c3eb14420490d62ee5c9bee33e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46425 } } }
I20260924 07:18:27.005918 5225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [sys.catalog]: This master's current role is: LEADER
W20260924 07:18:27.005951 5238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 07:18:27.005930 5224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e [sys.catalog]: This master's current role is: LEADER
W20260924 07:18:27.006016 5238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:18:27.006156 5240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:27.006309 5240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:27.008237 5240 catalog_manager.cc:1347] Generated new cluster ID: 1253c2dab20b471fbf298525fe22f01c
I20260924 07:18:27.008292 5240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:18:27 dist-test-slave-rc1w krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:27.012516 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.001571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46406 (local address 127.3.246.126:46425)
0924 07:18:27.001801 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.001820 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.001869 (+ 49us) server_negotiation.cc:407] Connection header received
0924 07:18:27.002400 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.002409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.002475 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.002632 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:27.003385 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.004736 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.005897 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.006500 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.006631 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.009203 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:27.009207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:27.009208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:27.009219 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:27.010966 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.011356 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.011359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.011359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.011460 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.011681 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.011682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.011682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.011914 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:27.012083 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.012178 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.012315 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260924 07:18:27.073747 5240 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:27.074334 5240 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:27.081828 5240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e: Generated new TSK 0
I20260924 07:18:27.082052 5240 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:18:27.117228 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46425
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:27.200613 5244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:27.200865 5244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:27.200907 5244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:27.202117 5244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:27.202188 5244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:27.202225 5244 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:18:27.202282 5244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:27.203766 5244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.246.126:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:27.204107 5244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:27.204350 5244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:27.209394 5252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:27.209385 5250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:27.209407 5249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:27.209614 5244 server_base.cc:1034] running on GCE node
I20260924 07:18:27.209824 5244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:27.210034 5244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:27.211174 5244 hybrid_clock.cc:648] HybridClock initialized: now 1790234307211152 us; error 37 us; skew 500 ppm
Sep 24 07:18:27 dist-test-slave-rc1w krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:27.213634 5244 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:18:27.214197 5244 webserver.cc:460] Webserver started at http://127.3.246.65:33585/ using document root <none> and password file <none>
I20260924 07:18:27.214432 5244 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:27.214493 5244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:27.214646 5244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:27.215724 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "2f6324b1eacd45fea800a90cec8843c3"
format_stamp: "Formatted at 2026-09-24 07:18:27 on dist-test-slave-rc1w"
server_key: "5259cc358c01bb65399abe3ad33b92af"
server_key_iv: "c920f638c4d03878dfdc903f2e4d65bf"
server_key_version: "encryptionkey@0"
I20260924 07:18:27.216089 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "2f6324b1eacd45fea800a90cec8843c3"
format_stamp: "Formatted at 2026-09-24 07:18:27 on dist-test-slave-rc1w"
server_key: "5259cc358c01bb65399abe3ad33b92af"
server_key_iv: "c920f638c4d03878dfdc903f2e4d65bf"
server_key_version: "encryptionkey@0"
I20260924 07:18:27.217401 5244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:27.218214 5259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:27.218428 5244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:18:27.218541 5244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "2f6324b1eacd45fea800a90cec8843c3"
format_stamp: "Formatted at 2026-09-24 07:18:27 on dist-test-slave-rc1w"
server_key: "5259cc358c01bb65399abe3ad33b92af"
server_key_iv: "c920f638c4d03878dfdc903f2e4d65bf"
server_key_version: "encryptionkey@0"
I20260924 07:18:27.218629 5244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:27.245751 5244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:27.247730 5244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:27.247896 5244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:27.248462 5244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:27.248524 5244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:27.248571 5244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:27.248598 5244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:27.255553 5244 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:42747
I20260924 07:18:27.255649 5374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:42747 every 8 connection(s)
I20260924 07:18:27.255956 5244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:18:27 dist-test-slave-rc1w krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:27.259485 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.249084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:37363 (local address 127.3.246.126:46425)
0924 07:18:27.249156 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.249158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.249365 (+ 207us) server_negotiation.cc:407] Connection header received
0924 07:18:27.250099 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.250101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.250117 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.250202 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:27.251089 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.251941 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.253507 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.253883 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.253969 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.256761 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:27.256762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:27.256762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:27.256770 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:27.258043 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.258490 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.258492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.258492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.258557 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.258826 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.258827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.258827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.258949 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:27.259023 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.259303 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.259395 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:18:27.259665 5268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.249045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46425 (local address 127.3.246.65:37363)
0924 07:18:27.249233 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.249274 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.249991 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.250309 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.250314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.250679 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:27.250974 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.250980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.252049 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.252050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.253410 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.253413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.253997 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.253998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.254184 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.254855 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:27.254872 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:27.256627 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:27.258139 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.258146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.258148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.258411 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.258626 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.258627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.258627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.258752 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.259025 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:27.259032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:27.259167 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.259282 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.259436 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260924 07:18:27.260212 5375 heartbeater.cc:347] Connected to a master server at 127.3.246.126:46425
I20260924 07:18:27.260331 5375 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:27.260510 5375 heartbeater.cc:511] Master 127.3.246.126:46425 requested a full tablet report, sending...
I20260924 07:18:27.261054 5172 ts_manager.cc:194] Registered new tserver with Master: 2f6324b1eacd45fea800a90cec8843c3 (127.3.246.65:42747)
I20260924 07:18:27.262032 5172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:37363
I20260924 07:18:27.262550 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 5244
I20260924 07:18:27.262647 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:27.262820 4057 external_mini_cluster.cc:1428] Setting key 7873e61fa62b914f13b09410f911b885
I20260924 07:18:27.267551 5381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.261287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46425 (local address 127.3.246.65:52879)
0924 07:18:27.261440 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.261452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.261512 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.261626 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.261626 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.261872 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:27.261929 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.261931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.262739 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.262740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.263333 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.263335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.263816 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.263817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.263877 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.264334 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:27.264338 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:27.264720 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:27.266617 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.266617 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.266618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.266780 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.266997 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.266997 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.266998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.267050 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.267321 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:27.267322 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:27.267359 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.267408 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.267484 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260924 07:18:27.267661 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.261363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:52879 (local address 127.3.246.126:46425)
0924 07:18:27.261414 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.261416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.261497 (+ 81us) server_negotiation.cc:407] Connection header received
0924 07:18:27.261566 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.261566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.261574 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.261614 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:27.262001 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.262569 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.263435 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.263745 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.263814 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.264808 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:27.264809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:27.264809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:27.264815 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:27.266535 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.266859 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.266860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.266860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.266926 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.267114 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.267115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.267115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.267258 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:27.267330 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.267448 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.267528 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:18:27.292191 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46425
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:27.371948 5384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:27.372141 5384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:27.372164 5384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:27.373327 5384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:27.373368 5384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:27.373389 5384 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:18:27.373405 5384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:27.374847 5384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.246.126:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:27.375070 5384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:27.375298 5384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:27.380354 5390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:27.380363 5389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:27.380445 5384 server_base.cc:1034] running on GCE node
W20260924 07:18:27.380396 5392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:27.380764 5384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:27.380970 5384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:27.382130 5384 hybrid_clock.cc:648] HybridClock initialized: now 1790234307382112 us; error 46 us; skew 500 ppm
Sep 24 07:18:27 dist-test-slave-rc1w krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:27.384557 5384 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:18:27.385040 5384 webserver.cc:460] Webserver started at http://127.3.246.66:36171/ using document root <none> and password file <none>
I20260924 07:18:27.385262 5384 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:27.385334 5384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:27.385486 5384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:27.386623 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "3e260e8a8c2c4630b472439eeba793b9"
format_stamp: "Formatted at 2026-09-24 07:18:27 on dist-test-slave-rc1w"
server_key: "48ea01ce4ed820487d1ff1f7a4f5a2f1"
server_key_iv: "20f9e31083682645145b9bb204f0a3cb"
server_key_version: "encryptionkey@0"
I20260924 07:18:27.386989 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "3e260e8a8c2c4630b472439eeba793b9"
format_stamp: "Formatted at 2026-09-24 07:18:27 on dist-test-slave-rc1w"
server_key: "48ea01ce4ed820487d1ff1f7a4f5a2f1"
server_key_iv: "20f9e31083682645145b9bb204f0a3cb"
server_key_version: "encryptionkey@0"
I20260924 07:18:27.388295 5384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:27.389072 5399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:27.389276 5384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:27.389379 5384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "3e260e8a8c2c4630b472439eeba793b9"
format_stamp: "Formatted at 2026-09-24 07:18:27 on dist-test-slave-rc1w"
server_key: "48ea01ce4ed820487d1ff1f7a4f5a2f1"
server_key_iv: "20f9e31083682645145b9bb204f0a3cb"
server_key_version: "encryptionkey@0"
I20260924 07:18:27.389463 5384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:27.399592 5384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:27.401537 5384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:27.401731 5384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:27.402364 5384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:27.402415 5384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:27.402446 5384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:27.402501 5384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:18:27 dist-test-slave-rc1w krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:27.409869 5384 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:46065
I20260924 07:18:27.409953 5514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:46065 every 8 connection(s)
I20260924 07:18:27.410267 5384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:27.413054 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.402798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:45333 (local address 127.3.246.126:46425)
0924 07:18:27.402871 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.402874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.403096 (+ 222us) server_negotiation.cc:407] Connection header received
0924 07:18:27.403648 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.403649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.403663 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.403750 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:27.404645 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.405509 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.406704 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.407046 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.407148 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.410232 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:27.410233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:27.410234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:27.410242 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:27.411603 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.412022 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.412024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.412024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.412093 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.412363 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.412364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.412365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.412492 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:27.412563 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.412877 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.412963 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:18:27.413236 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.402802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46425 (local address 127.3.246.66:45333)
0924 07:18:27.402961 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.403001 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.403542 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.403858 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.403861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.404216 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:27.404529 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.404535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.405616 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.405618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.406566 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.406569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.407166 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.407168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.407328 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.408048 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:27.408057 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:27.410094 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:27.411695 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.411703 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.411705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.411937 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.412174 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.412174 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.412175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.412286 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.412566 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:27.412574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:27.412719 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.412846 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.413008 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260924 07:18:27.413797 5515 heartbeater.cc:347] Connected to a master server at 127.3.246.126:46425
I20260924 07:18:27.413904 5515 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:27.414080 5515 heartbeater.cc:511] Master 127.3.246.126:46425 requested a full tablet report, sending...
I20260924 07:18:27.414534 5172 ts_manager.cc:194] Registered new tserver with Master: 3e260e8a8c2c4630b472439eeba793b9 (127.3.246.66:46065)
I20260924 07:18:27.415061 5172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:45333
I20260924 07:18:27.416957 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 5384
I20260924 07:18:27.417070 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:27.417279 4057 external_mini_cluster.cc:1428] Setting key 62c02be464f20a625735dbdd8edf88db
I20260924 07:18:27.421305 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.415068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:47689 (local address 127.3.246.126:46425)
0924 07:18:27.415113 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.415115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.415144 (+ 29us) server_negotiation.cc:407] Connection header received
0924 07:18:27.415259 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.415259 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.415271 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.415317 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:27.415645 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.416189 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.417207 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.417491 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.417581 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.418703 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:27.418705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:27.418705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:27.418712 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:27.420066 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.420427 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.420428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.420429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.420490 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.420718 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.420718 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.420719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.420883 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:27.420961 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.421114 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.421222 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:18:27.421345 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.414899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46425 (local address 127.3.246.66:47689)
0924 07:18:27.415078 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.415091 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.415158 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.415322 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.415322 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.415519 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:27.415568 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.415570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.416300 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.416301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.417055 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.417089 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.417579 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.417581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.417662 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.418280 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:27.418285 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:27.418573 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:27.420140 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.420141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.420142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.420348 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.420558 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.420559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.420559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.420618 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.420951 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:27.420952 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:27.420999 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.421089 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.421207 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:18:27.444922 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46425
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:27.528882 5524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:27.529098 5524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:27.529124 5524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:27.530414 5524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:27.530462 5524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:27.530483 5524 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:18:27.530499 5524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:27.531972 5524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.246.126:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:27.532222 5524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:27.532476 5524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:27.537470 5529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:27.537444 5532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:27.537621 5524 server_base.cc:1034] running on GCE node
W20260924 07:18:27.537451 5530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:27.537971 5524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:27.538241 5524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:27.539430 5524 hybrid_clock.cc:648] HybridClock initialized: now 1790234307539415 us; error 31 us; skew 500 ppm
Sep 24 07:18:27 dist-test-slave-rc1w krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:27.541990 5524 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:18:27.542686 5524 webserver.cc:460] Webserver started at http://127.3.246.67:33939/ using document root <none> and password file <none>
I20260924 07:18:27.542944 5524 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:27.543015 5524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:27.543167 5524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:27.544282 5524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "c09f7e44a1b543ada3834880ce215482"
format_stamp: "Formatted at 2026-09-24 07:18:27 on dist-test-slave-rc1w"
server_key: "94ffcab1e0bcf0ed624d815318b2b269"
server_key_iv: "8ad482fc60442293f44429f245320b8b"
server_key_version: "encryptionkey@0"
I20260924 07:18:27.544657 5524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "c09f7e44a1b543ada3834880ce215482"
format_stamp: "Formatted at 2026-09-24 07:18:27 on dist-test-slave-rc1w"
server_key: "94ffcab1e0bcf0ed624d815318b2b269"
server_key_iv: "8ad482fc60442293f44429f245320b8b"
server_key_version: "encryptionkey@0"
I20260924 07:18:27.546134 5524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 07:18:27.546870 5539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:27.547037 5524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:27.547143 5524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "c09f7e44a1b543ada3834880ce215482"
format_stamp: "Formatted at 2026-09-24 07:18:27 on dist-test-slave-rc1w"
server_key: "94ffcab1e0bcf0ed624d815318b2b269"
server_key_iv: "8ad482fc60442293f44429f245320b8b"
server_key_version: "encryptionkey@0"
I20260924 07:18:27.547222 5524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:27.582374 5524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:27.584358 5524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:27.584532 5524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:27.585044 5524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:27.585110 5524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:27.585156 5524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:27.585184 5524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:18:27 dist-test-slave-rc1w krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:27.592959 5524 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:36435
I20260924 07:18:27.593034 5654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:36435 every 8 connection(s)
I20260924 07:18:27.593449 5524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:27.595508 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.585665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:53885 (local address 127.3.246.126:46425)
0924 07:18:27.585723 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.585726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.586073 (+ 347us) server_negotiation.cc:407] Connection header received
0924 07:18:27.586613 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.586614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.586629 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.586696 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:27.587336 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.587854 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.589021 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.589238 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.589342 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.592153 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:27.592155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:27.592155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:27.592165 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:27.594042 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.594472 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.594473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.594474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.594542 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.594813 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.594814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.594814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.594942 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:27.595021 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.595314 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.595401 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:18:27.595700 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.585690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46425 (local address 127.3.246.67:53885)
0924 07:18:27.585932 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.585969 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.586502 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.586765 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.586768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.587040 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:27.587237 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.587241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.587955 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.587956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.588892 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.588895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.589354 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.589355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.589485 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.590063 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:27.590072 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:27.591990 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:27.594139 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.594145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.594147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.594381 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.594615 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.594615 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.594616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.594738 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.595012 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:27.595018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:27.595168 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.595279 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.595466 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
I20260924 07:18:27.596222 5655 heartbeater.cc:347] Connected to a master server at 127.3.246.126:46425
I20260924 07:18:27.596320 5655 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:27.596504 5655 heartbeater.cc:511] Master 127.3.246.126:46425 requested a full tablet report, sending...
I20260924 07:18:27.596942 5171 ts_manager.cc:194] Registered new tserver with Master: c09f7e44a1b543ada3834880ce215482 (127.3.246.67:36435)
I20260924 07:18:27.597651 5171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:53885
I20260924 07:18:27.600528 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 5524
I20260924 07:18:27.600625 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:27.600793 4057 external_mini_cluster.cc:1428] Setting key bed5e09bca96dac74867ab7932989843
I20260924 07:18:27.601955 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:18:27.603713 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.597602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:49109 (local address 127.3.246.126:46425)
0924 07:18:27.597663 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.597665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.597695 (+ 30us) server_negotiation.cc:407] Connection header received
0924 07:18:27.597724 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.597725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.597735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.597798 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:27.598428 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.598908 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.599672 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.599861 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.599924 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.601011 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:27.601013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:27.601013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:27.601021 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:27.602569 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.602930 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.602931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.602932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.602988 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.603199 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.603200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.603200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.603313 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:27.603389 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.603552 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.603622 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:18:27.603737 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.597375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46425 (local address 127.3.246.67:49109)
0924 07:18:27.597583 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.597597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.597655 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.597844 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.597845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.598240 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:27.598303 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.598306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.599044 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.599045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.599564 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.599565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.599936 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.599937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.599994 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.600518 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:27.600524 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:27.600922 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:27.602655 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.602656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.602657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.602840 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.603062 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.603063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.603063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.603124 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.603381 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:27.603382 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:27.603430 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.603513 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.603622 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
Sep 24 07:18:27 dist-test-slave-rc1w krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:18:27 dist-test-slave-rc1w krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:27.628265 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.619280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46422 (local address 127.3.246.126:46425)
0924 07:18:27.619359 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.619361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.619370 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:27.619397 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.619398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.619408 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.619470 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:27.620200 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.620794 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.621956 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.622214 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.622276 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.625198 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:27.625200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:27.625200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:27.625217 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:27.626986 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.627403 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.627404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.627405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.627471 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.627679 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.627680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.627680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.627807 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:27.627926 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.628064 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.628153 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:18:27.630245 5171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:27.631867 5171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:27.639035 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.634726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:46065 (local address 127.0.0.1:53096)
0924 07:18:27.635431 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.635442 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.635524 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.635755 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.635756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.635757 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:27.635800 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.635801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.636651 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.636652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.637791 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.637793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.638732 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.638732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.638807 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.638825 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.638899 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.638955 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":104,"threads_started":1}
I20260924 07:18:27.639081 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.634903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.3.246.66:46065)
0924 07:18:27.635096 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.635100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.635496 (+ 396us) server_negotiation.cc:407] Connection header received
0924 07:18:27.635619 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.635622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.635636 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.635689 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:27.635873 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.636559 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.637874 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.638633 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.638732 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.638800 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.638926 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.638963 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
I20260924 07:18:27.639832 5676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.634478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:42747 (local address 127.0.0.1:51378)
0924 07:18:27.634893 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.634939 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.635019 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.635215 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.635217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.635219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:27.635287 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.635289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.637778 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.637778 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.638553 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.638555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.639545 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.639546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.639620 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.639625 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.639689 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.639738 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":36,"thread_start_us":73,"threads_started":1}
I20260924 07:18:27.640138 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.634557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:36435 (local address 127.0.0.1:54904)
0924 07:18:27.634898 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.634938 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.635007 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.635331 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.635332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.635334 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:27.635378 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.635379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.637796 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.637797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.638966 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.638969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.639903 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.639904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.639975 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.639985 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.640030 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.640069 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":88,"threads_started":1}
I20260924 07:18:27.640216 5449 tablet_service.cc:1467] Processing CreateTablet for tablet 6682c87b49fb4edca98103fd0f5a6ec9 (DEFAULT_TABLE table=test-table [id=f2f67c3ca1714500931b0a49f90efda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:27.640401 5675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.634682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54904 (local address 127.3.246.67:36435)
0924 07:18:27.635176 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.635179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.635186 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:18:27.635216 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.635218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.635229 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.635336 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:27.635482 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.636546 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.639065 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.639754 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.639816 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.639866 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.640277 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.640320 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":110,"threads_started":1}
I20260924 07:18:27.640642 5449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6682c87b49fb4edca98103fd0f5a6ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:27.641040 5589 tablet_service.cc:1467] Processing CreateTablet for tablet 6682c87b49fb4edca98103fd0f5a6ec9 (DEFAULT_TABLE table=test-table [id=f2f67c3ca1714500931b0a49f90efda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:27.641350 5589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6682c87b49fb4edca98103fd0f5a6ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:27.643970 5679 tablet_bootstrap.cc:492] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482: Bootstrap starting.
I20260924 07:18:27.644090 5678 tablet_bootstrap.cc:492] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9: Bootstrap starting.
I20260924 07:18:27.644246 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.634620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.3.246.65:42747)
0924 07:18:27.634883 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.634888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.635009 (+ 121us) server_negotiation.cc:407] Connection header received
0924 07:18:27.635096 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.635098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.635114 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.635213 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:27.635370 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.636552 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.638647 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.639448 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.639529 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.639587 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.639713 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.644134 (+ 4421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":124,"threads_started":1}
I20260924 07:18:27.644827 5678 tablet_bootstrap.cc:654] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:27.644833 5679 tablet_bootstrap.cc:654] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:27.644912 5309 tablet_service.cc:1467] Processing CreateTablet for tablet 6682c87b49fb4edca98103fd0f5a6ec9 (DEFAULT_TABLE table=test-table [id=f2f67c3ca1714500931b0a49f90efda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:27.645206 5678 log.cc:824] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:27.645196 5679 log.cc:824] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:27.645269 5309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6682c87b49fb4edca98103fd0f5a6ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:27.646119 5678 tablet_bootstrap.cc:492] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9: No bootstrap required, opened a new log
I20260924 07:18:27.646241 5678 ts_tablet_manager.cc:1397] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:18:27.648391 5678 raft_consensus.cc:348] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.648577 5678 raft_consensus.cc:374] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:27.648617 5678 raft_consensus.cc:729] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e260e8a8c2c4630b472439eeba793b9, State: Initialized, Role: FOLLOWER
I20260924 07:18:27.648759 5678 consensus_queue.cc:260] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.649041 5678 ts_tablet_manager.cc:1428] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:27.649185 5515 heartbeater.cc:503] Master 127.3.246.126:46425 was elected leader, sending a full tablet report...
I20260924 07:18:27.649359 5682 tablet_bootstrap.cc:492] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3: Bootstrap starting.
I20260924 07:18:27.649492 5679 tablet_bootstrap.cc:492] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482: No bootstrap required, opened a new log
I20260924 07:18:27.649565 5679 ts_tablet_manager.cc:1397] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260924 07:18:27.650033 5682 tablet_bootstrap.cc:654] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:27.650583 5682 log.cc:824] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:27.651492 5682 tablet_bootstrap.cc:492] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3: No bootstrap required, opened a new log
I20260924 07:18:27.651575 5682 ts_tablet_manager.cc:1397] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:18:27.651839 5679 raft_consensus.cc:348] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.651968 5679 raft_consensus.cc:374] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:27.651989 5679 raft_consensus.cc:729] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09f7e44a1b543ada3834880ce215482, State: Initialized, Role: FOLLOWER
I20260924 07:18:27.652117 5679 consensus_queue.cc:260] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.652395 5679 ts_tablet_manager.cc:1428] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 07:18:27.652463 5655 heartbeater.cc:503] Master 127.3.246.126:46425 was elected leader, sending a full tablet report...
I20260924 07:18:27.653182 5682 raft_consensus.cc:348] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.653337 5682 raft_consensus.cc:374] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:27.653383 5682 raft_consensus.cc:729] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6324b1eacd45fea800a90cec8843c3, State: Initialized, Role: FOLLOWER
I20260924 07:18:27.653492 5682 consensus_queue.cc:260] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.653806 5682 ts_tablet_manager.cc:1428] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:18:27.653854 5375 heartbeater.cc:503] Master 127.3.246.126:46425 was elected leader, sending a full tablet report...
W20260924 07:18:27.660857 5516 tablet.cc:2369] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:18:27.748114 5686 raft_consensus.cc:482] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:27.748253 5686 raft_consensus.cc:504] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.748574 5686 leader_election.cc:288] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e260e8a8c2c4630b472439eeba793b9 (127.3.246.66:46065), c09f7e44a1b543ada3834880ce215482 (127.3.246.67:36435)
I20260924 07:18:27.752590 5268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.748861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:46065 (local address 127.3.246.65:34661)
0924 07:18:27.748937 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.748950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.749019 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.749224 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.749225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.749226 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:27.749342 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.749344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.750320 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.750322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.751477 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.751480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.752273 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.752274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.752366 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.752369 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.752473 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.752505 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":53}
I20260924 07:18:27.752611 5675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.748955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:33113 (local address 127.3.246.67:36435)
0924 07:18:27.749005 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.749007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.749126 (+ 119us) server_negotiation.cc:407] Connection header received
0924 07:18:27.749175 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.749176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.749187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.749242 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:27.749413 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.750233 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.751581 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.752144 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.752225 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.752234 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.752486 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.752535 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:18:27.752615 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.748989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:34661 (local address 127.3.246.66:46065)
0924 07:18:27.749041 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.749043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.749051 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:18:27.749091 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.749092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.749119 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.749201 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:27.749419 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.750206 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.751586 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.752153 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.752225 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.752234 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.752488 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.752535 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 07:18:27.752599 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.748861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:36435 (local address 127.3.246.65:33113)
0924 07:18:27.749052 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.749061 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.749122 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.749259 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.749260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.749261 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:27.749342 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.749344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.750320 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.750322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.751477 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.751480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.752273 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.752274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.752359 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.752363 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.752441 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.752486 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":232,"thread_start_us":87,"threads_started":1}
I20260924 07:18:27.753242 5466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6682c87b49fb4edca98103fd0f5a6ec9" candidate_uuid: "2f6324b1eacd45fea800a90cec8843c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e260e8a8c2c4630b472439eeba793b9" is_pre_election: true
I20260924 07:18:27.753234 5605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6682c87b49fb4edca98103fd0f5a6ec9" candidate_uuid: "2f6324b1eacd45fea800a90cec8843c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09f7e44a1b543ada3834880ce215482" is_pre_election: true
I20260924 07:18:27.753405 5605 raft_consensus.cc:2454] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6324b1eacd45fea800a90cec8843c3 in term 0.
I20260924 07:18:27.753417 5466 raft_consensus.cc:2454] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6324b1eacd45fea800a90cec8843c3 in term 0.
I20260924 07:18:27.753669 5260 leader_election.cc:302] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6324b1eacd45fea800a90cec8843c3, 3e260e8a8c2c4630b472439eeba793b9; no voters:
I20260924 07:18:27.753882 5686 raft_consensus.cc:2781] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:27.754006 5686 raft_consensus.cc:482] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:27.754046 5686 raft_consensus.cc:3037] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:27.754776 5686 raft_consensus.cc:504] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.754935 5686 leader_election.cc:288] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [CANDIDATE]: Term 1 election: Requested vote from peers 3e260e8a8c2c4630b472439eeba793b9 (127.3.246.66:46065), c09f7e44a1b543ada3834880ce215482 (127.3.246.67:36435)
I20260924 07:18:27.755124 5605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6682c87b49fb4edca98103fd0f5a6ec9" candidate_uuid: "2f6324b1eacd45fea800a90cec8843c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09f7e44a1b543ada3834880ce215482"
I20260924 07:18:27.755125 5466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6682c87b49fb4edca98103fd0f5a6ec9" candidate_uuid: "2f6324b1eacd45fea800a90cec8843c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e260e8a8c2c4630b472439eeba793b9"
I20260924 07:18:27.755208 5605 raft_consensus.cc:3037] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:27.755208 5466 raft_consensus.cc:3037] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:27.756090 5466 raft_consensus.cc:2454] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6324b1eacd45fea800a90cec8843c3 in term 1.
I20260924 07:18:27.756090 5605 raft_consensus.cc:2454] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6324b1eacd45fea800a90cec8843c3 in term 1.
I20260924 07:18:27.756270 5260 leader_election.cc:302] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6324b1eacd45fea800a90cec8843c3, 3e260e8a8c2c4630b472439eeba793b9; no voters:
I20260924 07:18:27.756441 5686 raft_consensus.cc:2781] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [term 1 FOLLOWER]: Leader election won for term 1
W20260924 07:18:27.756614 5376 tablet.cc:2369] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:18:27.756633 5686 raft_consensus.cc:686] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [term 1 LEADER]: Becoming Leader. State: Replica: 2f6324b1eacd45fea800a90cec8843c3, State: Running, Role: LEADER
I20260924 07:18:27.756742 5686 consensus_queue.cc:237] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.757776 5172 catalog_manager.cc:5495] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f6324b1eacd45fea800a90cec8843c3 (127.3.246.65). New cstate: current_term: 1 leader_uuid: "2f6324b1eacd45fea800a90cec8843c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } health_report { overall_health: UNKNOWN } } }
I20260924 07:18:27.784246 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.781823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.3.246.65:42747)
0924 07:18:27.781915 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.781917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.781927 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:18:27.781965 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.781965 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.781982 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.782043 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:27.782282 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.782764 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.783559 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.783742 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.783793 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.783973 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:27.784061 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:27.784118 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.784164 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.784186 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 07:18:27.786108 5466 raft_consensus.cc:1261] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9 [term 1 FOLLOWER]: Refusing update from remote peer 2f6324b1eacd45fea800a90cec8843c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:27.786108 5605 raft_consensus.cc:1261] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482 [term 1 FOLLOWER]: Refusing update from remote peer 2f6324b1eacd45fea800a90cec8843c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:27.786504 5688 consensus_queue.cc:1035] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:27.786720 5686 consensus_queue.cc:1035] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:27.789283 5693 mvcc.cc:205] Tried to move back new op lower bound from 7332799724688642048 to 7332799724572196864. Current Snapshot: MvccSnapshot[applied={T|T < 7332799724688642048}]
I20260924 07:18:27.789896 5692 mvcc.cc:205] Tried to move back new op lower bound from 7332799724688642048 to 7332799724572196864. Current Snapshot: MvccSnapshot[applied={T|T < 7332799724688642048}]
I20260924 07:18:27.792668 5172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:27.792774 5172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:27.793650 5172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e: Sending DeleteTablet for 3 replicas of tablet 6682c87b49fb4edca98103fd0f5a6ec9
I20260924 07:18:27.794020 5589 tablet_service.cc:1514] Processing DeleteTablet for tablet 6682c87b49fb4edca98103fd0f5a6ec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:27 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:54904
I20260924 07:18:27.794045 5309 tablet_service.cc:1514] Processing DeleteTablet for tablet 6682c87b49fb4edca98103fd0f5a6ec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:27 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:51378
I20260924 07:18:27.794322 5449 tablet_service.cc:1514] Processing DeleteTablet for tablet 6682c87b49fb4edca98103fd0f5a6ec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:27 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:53096
I20260924 07:18:27.794520 5172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:27.794659 5172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:27.794776 5702 tablet_replica.cc:330] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9: stopping tablet replica
I20260924 07:18:27.794912 5702 raft_consensus.cc:2229] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:27.795055 5702 raft_consensus.cc:2258] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:27.795522 5702 ts_tablet_manager.cc:1905] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:27.795713 5700 tablet_replica.cc:330] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3: stopping tablet replica
I20260924 07:18:27.795935 5589 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1864ef4b614db0beb29866c7dd7e9b (DEFAULT_TABLE table=test-table [id=cd678e7b72c74229b2e28aaec52ce163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:27.795979 5309 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1864ef4b614db0beb29866c7dd7e9b (DEFAULT_TABLE table=test-table [id=cd678e7b72c74229b2e28aaec52ce163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:27.796070 5589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1864ef4b614db0beb29866c7dd7e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:27.796087 5309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1864ef4b614db0beb29866c7dd7e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:27.797317 5449 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1864ef4b614db0beb29866c7dd7e9b (DEFAULT_TABLE table=test-table [id=cd678e7b72c74229b2e28aaec52ce163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:27.797549 5702 ts_tablet_manager.cc:1918] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:27.797686 5702 log.cc:1196] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/6682c87b49fb4edca98103fd0f5a6ec9
I20260924 07:18:27.798002 5679 tablet_bootstrap.cc:492] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482: Bootstrap starting.
I20260924 07:18:27.798139 5682 tablet_bootstrap.cc:492] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3: Bootstrap starting.
I20260924 07:18:27.798183 5449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1864ef4b614db0beb29866c7dd7e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:27.798412 5679 tablet_bootstrap.cc:654] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:27.798542 5682 tablet_bootstrap.cc:654] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:27.798892 5702 ts_tablet_manager.cc:1939] T 6682c87b49fb4edca98103fd0f5a6ec9 P 3e260e8a8c2c4630b472439eeba793b9: Deleting consensus metadata
I20260924 07:18:27.798990 5679 tablet_bootstrap.cc:492] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482: No bootstrap required, opened a new log
I20260924 07:18:27.799031 5679 ts_tablet_manager.cc:1397] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:27.799190 5679 raft_consensus.cc:348] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.799263 5679 raft_consensus.cc:374] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:27.799281 5679 raft_consensus.cc:729] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09f7e44a1b543ada3834880ce215482, State: Initialized, Role: FOLLOWER
I20260924 07:18:27.799341 5679 consensus_queue.cc:260] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.799448 5679 ts_tablet_manager.cc:1428] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:27.799445 5678 tablet_bootstrap.cc:492] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9: Bootstrap starting.
I20260924 07:18:27.799455 5156 catalog_manager.cc:4841] TS 3e260e8a8c2c4630b472439eeba793b9 (127.3.246.66:46065): tablet 6682c87b49fb4edca98103fd0f5a6ec9 (table test-table [id=f2f67c3ca1714500931b0a49f90efda1]) successfully deleted
I20260924 07:18:27.799880 5678 tablet_bootstrap.cc:654] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:27.800127 5700 raft_consensus.cc:2229] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:27.800251 5700 raft_consensus.cc:2258] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:27.800513 5678 tablet_bootstrap.cc:492] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9: No bootstrap required, opened a new log
I20260924 07:18:27.800576 5678 ts_tablet_manager.cc:1397] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:27.800680 5700 ts_tablet_manager.cc:1905] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:27.800719 5678 raft_consensus.cc:348] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.800765 5678 raft_consensus.cc:374] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:27.800778 5678 raft_consensus.cc:729] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e260e8a8c2c4630b472439eeba793b9, State: Initialized, Role: FOLLOWER
I20260924 07:18:27.800834 5678 consensus_queue.cc:260] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.800921 5678 ts_tablet_manager.cc:1428] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:27.801801 5701 tablet_replica.cc:330] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482: stopping tablet replica
I20260924 07:18:27.801949 5701 raft_consensus.cc:2229] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:27.802086 5701 raft_consensus.cc:2258] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:27.802372 5700 ts_tablet_manager.cc:1918] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:27.802449 5700 log.cc:1196] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/wal/wals/6682c87b49fb4edca98103fd0f5a6ec9
I20260924 07:18:27.802634 5701 ts_tablet_manager.cc:1905] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:27.802744 5700 ts_tablet_manager.cc:1939] T 6682c87b49fb4edca98103fd0f5a6ec9 P 2f6324b1eacd45fea800a90cec8843c3: Deleting consensus metadata
I20260924 07:18:27.803227 5156 catalog_manager.cc:4841] TS 2f6324b1eacd45fea800a90cec8843c3 (127.3.246.65:42747): tablet 6682c87b49fb4edca98103fd0f5a6ec9 (table test-table [id=f2f67c3ca1714500931b0a49f90efda1]) successfully deleted
I20260924 07:18:27.804176 5701 ts_tablet_manager.cc:1918] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:27.804255 5701 log.cc:1196] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/6682c87b49fb4edca98103fd0f5a6ec9
I20260924 07:18:27.804504 5701 ts_tablet_manager.cc:1939] T 6682c87b49fb4edca98103fd0f5a6ec9 P c09f7e44a1b543ada3834880ce215482: Deleting consensus metadata
I20260924 07:18:27.804858 5159 catalog_manager.cc:4841] TS c09f7e44a1b543ada3834880ce215482 (127.3.246.67:36435): tablet 6682c87b49fb4edca98103fd0f5a6ec9 (table test-table [id=f2f67c3ca1714500931b0a49f90efda1]) successfully deleted
I20260924 07:18:27.810101 5682 tablet_bootstrap.cc:492] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3: No bootstrap required, opened a new log
I20260924 07:18:27.810155 5682 ts_tablet_manager.cc:1397] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.001s
I20260924 07:18:27.810334 5682 raft_consensus.cc:348] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.810407 5682 raft_consensus.cc:374] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:27.810421 5682 raft_consensus.cc:729] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6324b1eacd45fea800a90cec8843c3, State: Initialized, Role: FOLLOWER
I20260924 07:18:27.810480 5682 consensus_queue.cc:260] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.810580 5682 ts_tablet_manager.cc:1428] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260924 07:18:27.844050 5656 tablet.cc:2369] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:18:27.920442 5683 raft_consensus.cc:482] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:27.920581 5683 raft_consensus.cc:504] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.920991 5683 leader_election.cc:288] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f6324b1eacd45fea800a90cec8843c3 (127.3.246.65:42747), c09f7e44a1b543ada3834880ce215482 (127.3.246.67:36435)
I20260924 07:18:27.924054 5675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.921332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:50095 (local address 127.3.246.67:36435)
0924 07:18:27.921377 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.921380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.921450 (+ 70us) server_negotiation.cc:407] Connection header received
0924 07:18:27.921662 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.921664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.921701 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.921752 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:27.921928 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.922435 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.923355 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.923709 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.923793 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.923802 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.923957 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.923990 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:18:27.924072 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.921232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:36435 (local address 127.3.246.66:50095)
0924 07:18:27.921373 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.921385 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.921588 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.921774 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.921775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.921777 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:27.921850 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.921851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.922499 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.922500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.923254 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.923256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.923791 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.923792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.923875 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.923879 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.923940 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.923983 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260924 07:18:27.924144 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.921327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:54079 (local address 127.3.246.65:42747)
0924 07:18:27.921377 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.921379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.921450 (+ 71us) server_negotiation.cc:407] Connection header received
0924 07:18:27.921541 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.921542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.921552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.921625 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:27.921801 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.922362 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.923409 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.923783 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.923877 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.923885 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.924040 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.924075 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:18:27.924171 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.921234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:42747 (local address 127.3.246.66:54079)
0924 07:18:27.921391 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.921402 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.921466 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.921630 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.921630 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.921633 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:27.921725 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.921727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.922438 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.922439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.923320 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.923322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.923878 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.923879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.923950 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.923953 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.924032 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.924075 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260924 07:18:27.924425 5605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1864ef4b614db0beb29866c7dd7e9b" candidate_uuid: "3e260e8a8c2c4630b472439eeba793b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09f7e44a1b543ada3834880ce215482" is_pre_election: true
I20260924 07:18:27.924463 5329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1864ef4b614db0beb29866c7dd7e9b" candidate_uuid: "3e260e8a8c2c4630b472439eeba793b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6324b1eacd45fea800a90cec8843c3" is_pre_election: true
I20260924 07:18:27.924528 5605 raft_consensus.cc:2454] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e260e8a8c2c4630b472439eeba793b9 in term 0.
I20260924 07:18:27.924592 5329 raft_consensus.cc:2454] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e260e8a8c2c4630b472439eeba793b9 in term 0.
I20260924 07:18:27.924705 5403 leader_election.cc:302] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e260e8a8c2c4630b472439eeba793b9, c09f7e44a1b543ada3834880ce215482; no voters:
I20260924 07:18:27.924849 5683 raft_consensus.cc:2781] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:27.924890 5683 raft_consensus.cc:482] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:27.924911 5683 raft_consensus.cc:3037] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:27.925562 5683 raft_consensus.cc:504] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.925735 5683 leader_election.cc:288] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [CANDIDATE]: Term 1 election: Requested vote from peers 2f6324b1eacd45fea800a90cec8843c3 (127.3.246.65:42747), c09f7e44a1b543ada3834880ce215482 (127.3.246.67:36435)
I20260924 07:18:27.925868 5329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1864ef4b614db0beb29866c7dd7e9b" candidate_uuid: "3e260e8a8c2c4630b472439eeba793b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6324b1eacd45fea800a90cec8843c3"
I20260924 07:18:27.925894 5605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1864ef4b614db0beb29866c7dd7e9b" candidate_uuid: "3e260e8a8c2c4630b472439eeba793b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09f7e44a1b543ada3834880ce215482"
I20260924 07:18:27.925941 5329 raft_consensus.cc:3037] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:27.925952 5605 raft_consensus.cc:3037] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:27.926687 5605 raft_consensus.cc:2454] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e260e8a8c2c4630b472439eeba793b9 in term 1.
I20260924 07:18:27.926752 5329 raft_consensus.cc:2454] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e260e8a8c2c4630b472439eeba793b9 in term 1.
I20260924 07:18:27.926898 5403 leader_election.cc:302] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e260e8a8c2c4630b472439eeba793b9, c09f7e44a1b543ada3834880ce215482; no voters:
I20260924 07:18:27.927018 5683 raft_consensus.cc:2781] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:27.927119 5683 raft_consensus.cc:686] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [term 1 LEADER]: Becoming Leader. State: Replica: 3e260e8a8c2c4630b472439eeba793b9, State: Running, Role: LEADER
I20260924 07:18:27.927203 5683 consensus_queue.cc:237] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.927894 5171 catalog_manager.cc:5495] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e260e8a8c2c4630b472439eeba793b9 (127.3.246.66). New cstate: current_term: 1 leader_uuid: "3e260e8a8c2c4630b472439eeba793b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } health_report { overall_health: UNKNOWN } } }
Sep 24 07:18:27 dist-test-slave-rc1w krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:27.969202 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.962425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46425 (local address 127.0.0.1:46436)
0924 07:18:27.962631 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:27.962642 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:27.962740 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:27.962890 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:27.962891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:27.963207 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:27.963253 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.963256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.963952 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.963954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.964496 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:27.964497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.964844 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.964844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:27.964907 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.965410 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:27.965414 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:27.966814 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:27.968150 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.968157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.968159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.968353 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.968581 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:27.968582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:27.968582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:27.968637 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:27.968891 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:27.968898 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:27.968943 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 07:18:27.969015 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:18:27.969125 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
I20260924 07:18:27.969285 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:27.962529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.3.246.126:46425)
0924 07:18:27.962622 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:27.962624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:27.962720 (+ 96us) server_negotiation.cc:407] Connection header received
0924 07:18:27.962826 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:27.962827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:27.962838 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:27.962886 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:27.963369 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.963862 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.964581 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:27.964766 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:27.964844 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:27.966911 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:27.966913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:27.966913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:27.966919 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:27.968080 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.968454 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.968456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.968456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.968511 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:27.968695 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:27.968696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:27.968696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:27.968809 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:27.968862 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:27.969079 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 07:18:27.969177 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":61}
I20260924 07:18:27.971418 5171 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:46436:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:18:27.971601 5171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:27.973120 5309 tablet_service.cc:1467] Processing CreateTablet for tablet f28c341c5164425db53c4a7117aeaebf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8ae0a469aee84d04a2c0dde62cac5641]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:18:27.973122 5449 tablet_service.cc:1467] Processing CreateTablet for tablet f28c341c5164425db53c4a7117aeaebf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8ae0a469aee84d04a2c0dde62cac5641]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:18:27.973234 5589 tablet_service.cc:1467] Processing CreateTablet for tablet f28c341c5164425db53c4a7117aeaebf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8ae0a469aee84d04a2c0dde62cac5641]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:18:27.973299 5449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28c341c5164425db53c4a7117aeaebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:27.973301 5309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28c341c5164425db53c4a7117aeaebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:27.973363 5589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28c341c5164425db53c4a7117aeaebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:27.974635 5682 tablet_bootstrap.cc:492] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3: Bootstrap starting.
I20260924 07:18:27.974817 5679 tablet_bootstrap.cc:492] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482: Bootstrap starting.
I20260924 07:18:27.975019 5682 tablet_bootstrap.cc:654] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:27.975239 5679 tablet_bootstrap.cc:654] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:27.975584 5678 tablet_bootstrap.cc:492] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9: Bootstrap starting.
I20260924 07:18:27.975652 5682 tablet_bootstrap.cc:492] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3: No bootstrap required, opened a new log
I20260924 07:18:27.975687 5682 ts_tablet_manager.cc:1397] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:27.975826 5682 raft_consensus.cc:348] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.975905 5682 raft_consensus.cc:374] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:27.975930 5682 raft_consensus.cc:729] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6324b1eacd45fea800a90cec8843c3, State: Initialized, Role: FOLLOWER
I20260924 07:18:27.975983 5682 consensus_queue.cc:260] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.976027 5678 tablet_bootstrap.cc:654] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:27.976104 5682 ts_tablet_manager.cc:1428] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:27.976135 5688 tablet_replica.cc:439] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } } }
I20260924 07:18:27.976235 5688 tablet_replica.cc:442] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:27.976605 5679 tablet_bootstrap.cc:492] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482: No bootstrap required, opened a new log
I20260924 07:18:27.976655 5679 ts_tablet_manager.cc:1397] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:18:27.976861 5679 raft_consensus.cc:348] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.976940 5679 raft_consensus.cc:374] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:27.976959 5679 raft_consensus.cc:729] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09f7e44a1b543ada3834880ce215482, State: Initialized, Role: FOLLOWER
I20260924 07:18:27.977010 5679 consensus_queue.cc:260] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.977113 5679 ts_tablet_manager.cc:1428] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:27.977190 5685 tablet_replica.cc:439] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } } }
I20260924 07:18:27.977305 5685 tablet_replica.cc:442] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:27.977406 5678 tablet_bootstrap.cc:492] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9: No bootstrap required, opened a new log
I20260924 07:18:27.977455 5678 ts_tablet_manager.cc:1397] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:27.977633 5678 raft_consensus.cc:348] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.977735 5678 raft_consensus.cc:374] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:27.977769 5678 raft_consensus.cc:729] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e260e8a8c2c4630b472439eeba793b9, State: Initialized, Role: FOLLOWER
I20260924 07:18:27.977828 5678 consensus_queue.cc:260] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:27.977942 5678 ts_tablet_manager.cc:1428] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:27.977980 5683 tablet_replica.cc:439] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } } }
I20260924 07:18:27.978047 5683 tablet_replica.cc:442] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:28.301164 5688 raft_consensus.cc:482] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:28.301337 5688 raft_consensus.cc:504] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:28.301558 5688 leader_election.cc:288] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e260e8a8c2c4630b472439eeba793b9 (127.3.246.66:46065), c09f7e44a1b543ada3834880ce215482 (127.3.246.67:36435)
I20260924 07:18:28.301853 5605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28c341c5164425db53c4a7117aeaebf" candidate_uuid: "2f6324b1eacd45fea800a90cec8843c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09f7e44a1b543ada3834880ce215482" is_pre_election: true
I20260924 07:18:28.301875 5466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28c341c5164425db53c4a7117aeaebf" candidate_uuid: "2f6324b1eacd45fea800a90cec8843c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e260e8a8c2c4630b472439eeba793b9" is_pre_election: true
I20260924 07:18:28.301965 5605 raft_consensus.cc:2454] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6324b1eacd45fea800a90cec8843c3 in term 0.
I20260924 07:18:28.301965 5466 raft_consensus.cc:2454] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6324b1eacd45fea800a90cec8843c3 in term 0.
I20260924 07:18:28.302129 5263 leader_election.cc:302] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6324b1eacd45fea800a90cec8843c3, c09f7e44a1b543ada3834880ce215482; no voters:
I20260924 07:18:28.302237 5688 raft_consensus.cc:2781] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:28.302280 5688 raft_consensus.cc:482] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:28.302313 5688 raft_consensus.cc:3037] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:28.303128 5688 raft_consensus.cc:504] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:28.303273 5688 leader_election.cc:288] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [CANDIDATE]: Term 1 election: Requested vote from peers 3e260e8a8c2c4630b472439eeba793b9 (127.3.246.66:46065), c09f7e44a1b543ada3834880ce215482 (127.3.246.67:36435)
I20260924 07:18:28.303491 5605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28c341c5164425db53c4a7117aeaebf" candidate_uuid: "2f6324b1eacd45fea800a90cec8843c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09f7e44a1b543ada3834880ce215482"
I20260924 07:18:28.303467 5466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28c341c5164425db53c4a7117aeaebf" candidate_uuid: "2f6324b1eacd45fea800a90cec8843c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e260e8a8c2c4630b472439eeba793b9"
I20260924 07:18:28.303566 5605 raft_consensus.cc:3037] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:28.303575 5466 raft_consensus.cc:3037] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:28.304221 5605 raft_consensus.cc:2454] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6324b1eacd45fea800a90cec8843c3 in term 1.
I20260924 07:18:28.304252 5466 raft_consensus.cc:2454] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6324b1eacd45fea800a90cec8843c3 in term 1.
I20260924 07:18:28.304417 5263 leader_election.cc:302] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6324b1eacd45fea800a90cec8843c3, c09f7e44a1b543ada3834880ce215482; no voters:
I20260924 07:18:28.304543 5688 raft_consensus.cc:2781] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:28.304680 5688 raft_consensus.cc:686] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [term 1 LEADER]: Becoming Leader. State: Replica: 2f6324b1eacd45fea800a90cec8843c3, State: Running, Role: LEADER
I20260924 07:18:28.304752 5688 consensus_queue.cc:237] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } }
I20260924 07:18:28.305011 5719 tablet_replica.cc:439] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3: TxnStatusTablet state changed. Reason: New leader 2f6324b1eacd45fea800a90cec8843c3. Latest consensus state: current_term: 1 leader_uuid: "2f6324b1eacd45fea800a90cec8843c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } } }
I20260924 07:18:28.305104 5719 tablet_replica.cc:442] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:18:28.305250 5721 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:18:28.305388 5721 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:18:28.305446 5171 catalog_manager.cc:5495] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f6324b1eacd45fea800a90cec8843c3 (127.3.246.65). New cstate: current_term: 1 leader_uuid: "2f6324b1eacd45fea800a90cec8843c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } health_report { overall_health: UNKNOWN } } }
I20260924 07:18:28.332573 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:28.329621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:42747 (local address 127.0.0.1:45994)
0924 07:18:28.329744 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:28.329757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:28.329851 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:28.330032 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:28.330033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:28.330034 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:18:28.330082 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:28.330084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.331034 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.331034 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:28.331614 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:28.331616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.332100 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.332101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:28.332157 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:28.332172 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:18:28.332400 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:18:28.332411 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 07:18:28.332460 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 07:18:28.332508 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 07:18:28.332588 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:28.329743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.3.246.65:42747)
0924 07:18:28.329888 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:28.329890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:28.329901 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:18:28.329929 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:28.329931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:28.329943 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:28.330014 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:28.330150 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.330938 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.331703 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.332019 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.332090 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:28.332259 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:28.332297 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:28.332352 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:28.332485 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 07:18:28.332508 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260924 07:18:28.333832 5605 raft_consensus.cc:1261] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482 [term 1 FOLLOWER]: Refusing update from remote peer 2f6324b1eacd45fea800a90cec8843c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:28.333874 5466 raft_consensus.cc:1261] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9 [term 1 FOLLOWER]: Refusing update from remote peer 2f6324b1eacd45fea800a90cec8843c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:28.334048 5719 consensus_queue.cc:1035] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:28.334136 5688 consensus_queue.cc:1035] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:28.334703 5683 tablet_replica.cc:439] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9: TxnStatusTablet state changed. Reason: New leader 2f6324b1eacd45fea800a90cec8843c3. Latest consensus state: current_term: 1 leader_uuid: "2f6324b1eacd45fea800a90cec8843c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } } }
I20260924 07:18:28.334796 5683 tablet_replica.cc:442] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:28.334745 5685 tablet_replica.cc:439] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482: TxnStatusTablet state changed. Reason: New leader 2f6324b1eacd45fea800a90cec8843c3. Latest consensus state: current_term: 1 leader_uuid: "2f6324b1eacd45fea800a90cec8843c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } } }
I20260924 07:18:28.334833 5685 tablet_replica.cc:442] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:28.335891 5688 tablet_replica.cc:439] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f6324b1eacd45fea800a90cec8843c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } } }
I20260924 07:18:28.335975 5688 tablet_replica.cc:442] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:18:28.335928 5719 tablet_replica.cc:439] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f6324b1eacd45fea800a90cec8843c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } } }
I20260924 07:18:28.336009 5719 tablet_replica.cc:442] T f28c341c5164425db53c4a7117aeaebf P 2f6324b1eacd45fea800a90cec8843c3: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:18:28.335985 5683 tablet_replica.cc:439] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f6324b1eacd45fea800a90cec8843c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } } }
I20260924 07:18:28.336057 5683 tablet_replica.cc:442] T f28c341c5164425db53c4a7117aeaebf P 3e260e8a8c2c4630b472439eeba793b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:28.336061 5724 mvcc.cc:205] Tried to move back new op lower bound from 7332799726933647360 to 7332799726816817152. Current Snapshot: MvccSnapshot[applied={T|T < 7332799726933647360}]
I20260924 07:18:28.336450 5685 tablet_replica.cc:439] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f6324b1eacd45fea800a90cec8843c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 } } peers { permanent_uuid: "3e260e8a8c2c4630b472439eeba793b9" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 46065 } } peers { permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 } } }
I20260924 07:18:28.336526 5685 tablet_replica.cc:442] T f28c341c5164425db53c4a7117aeaebf P c09f7e44a1b543ada3834880ce215482: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:28.340828 5735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:28.338229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.3.246.66:46065)
0924 07:18:28.338331 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:28.338333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:28.338341 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:18:28.338436 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:28.338437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:28.338458 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:28.338553 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:28.338713 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.339187 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.339942 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.340217 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.340293 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:28.340472 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:28.340585 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:28.340668 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:28.340710 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 07:18:28.340729 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":60,"threads_started":1}
I20260924 07:18:28.346359 5738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:28.343575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:42747 (local address 127.3.246.66:54343)
0924 07:18:28.343742 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:28.343755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:28.343823 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:28.343968 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:28.343968 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:28.343970 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:18:28.344004 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:28.344005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.344875 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.344876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:28.345486 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:28.345488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.345971 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.345972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:28.346036 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:28.346049 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:18:28.346226 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:18:28.346227 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 07:18:28.346256 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 07:18:28.346292 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260924 07:18:28.346354 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:28.343773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:54343 (local address 127.3.246.65:42747)
0924 07:18:28.343858 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:28.343860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:28.343869 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:28.343890 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:28.343891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:28.343901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:28.343968 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:28.344043 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.344774 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.345570 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.345884 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.345955 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:28.346130 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:28.346164 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:28.346226 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:28.346262 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 07:18:28.346283 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:18:28.348942 5329 raft_consensus.cc:1261] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3 [term 1 FOLLOWER]: Refusing update from remote peer 3e260e8a8c2c4630b472439eeba793b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:28.349138 5605 raft_consensus.cc:1261] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482 [term 1 FOLLOWER]: Refusing update from remote peer 3e260e8a8c2c4630b472439eeba793b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:28.349231 5708 consensus_queue.cc:1035] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6324b1eacd45fea800a90cec8843c3" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:28.349330 5683 consensus_queue.cc:1035] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c09f7e44a1b543ada3834880ce215482" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:28.357577 5735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:28.355041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:45061 (local address 127.3.246.66:46065)
0924 07:18:28.355110 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:28.355112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:28.355229 (+ 117us) server_negotiation.cc:407] Connection header received
0924 07:18:28.355298 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:28.355298 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:28.355307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:28.355346 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:28.355469 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.355909 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.356998 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.357174 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.357226 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:28.357400 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:28.357425 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:28.357461 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:28.357513 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 07:18:28.357528 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:18:28.357632 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:28.355002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:46065 (local address 127.3.246.65:45061)
0924 07:18:28.355169 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:28.355180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:28.355241 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:28.355358 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:28.355358 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:28.355360 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:18:28.355416 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:28.355418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.355977 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.355978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:28.356917 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:28.356920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.357249 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.357250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:28.357308 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:28.357323 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:18:28.357463 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:18:28.357464 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 07:18:28.357511 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 07:18:28.357554 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260924 07:18:28.366855 5172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:28.366981 5172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:28.367610 5172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 910684c3eb14420490d62ee5c9bee33e: Sending DeleteTablet for 3 replicas of tablet 5f1864ef4b614db0beb29866c7dd7e9b
I20260924 07:18:28.367960 5589 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f1864ef4b614db0beb29866c7dd7e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:28 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:54904
I20260924 07:18:28.368187 5745 tablet_replica.cc:330] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482: stopping tablet replica
I20260924 07:18:28.368258 5449 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f1864ef4b614db0beb29866c7dd7e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:28 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:53096
I20260924 07:18:28.368294 5745 raft_consensus.cc:2229] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:28.368341 5745 raft_consensus.cc:2258] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:28.368484 5309 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f1864ef4b614db0beb29866c7dd7e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:28 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:51378
I20260924 07:18:28.369526 5745 ts_tablet_manager.cc:1905] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:28.371284 5745 ts_tablet_manager.cc:1918] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:28.371348 5745 log.cc:1196] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/5f1864ef4b614db0beb29866c7dd7e9b
I20260924 07:18:28.371560 5745 ts_tablet_manager.cc:1939] T 5f1864ef4b614db0beb29866c7dd7e9b P c09f7e44a1b543ada3834880ce215482: Deleting consensus metadata
I20260924 07:18:28.371940 5159 catalog_manager.cc:4841] TS c09f7e44a1b543ada3834880ce215482 (127.3.246.67:36435): tablet 5f1864ef4b614db0beb29866c7dd7e9b (table test-table [id=cd678e7b72c74229b2e28aaec52ce163]) successfully deleted
I20260924 07:18:28.372066 5746 tablet_replica.cc:330] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9: stopping tablet replica
I20260924 07:18:28.372131 5746 raft_consensus.cc:2229] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:28.372234 5746 raft_consensus.cc:2258] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:28.372488 5746 ts_tablet_manager.cc:1905] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:28.374058 5746 ts_tablet_manager.cc:1918] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:28.374207 5747 tablet_replica.cc:330] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3: stopping tablet replica
I20260924 07:18:28.374297 5747 raft_consensus.cc:2229] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:28.374408 5747 raft_consensus.cc:2258] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:28.374725 5747 ts_tablet_manager.cc:1905] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:28.376379 5747 ts_tablet_manager.cc:1918] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:28.376497 5746 log.cc:1196] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/5f1864ef4b614db0beb29866c7dd7e9b
I20260924 07:18:28.376806 5746 ts_tablet_manager.cc:1939] T 5f1864ef4b614db0beb29866c7dd7e9b P 3e260e8a8c2c4630b472439eeba793b9: Deleting consensus metadata
I20260924 07:18:28.377182 5156 catalog_manager.cc:4841] TS 3e260e8a8c2c4630b472439eeba793b9 (127.3.246.66:46065): tablet 5f1864ef4b614db0beb29866c7dd7e9b (table test-table [id=cd678e7b72c74229b2e28aaec52ce163]) successfully deleted
I20260924 07:18:28.377359 5747 log.cc:1196] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234301618289-4057-0/minicluster-data/ts-0/wal/wals/5f1864ef4b614db0beb29866c7dd7e9b
I20260924 07:18:28.377645 5747 ts_tablet_manager.cc:1939] T 5f1864ef4b614db0beb29866c7dd7e9b P 2f6324b1eacd45fea800a90cec8843c3: Deleting consensus metadata
I20260924 07:18:28.378082 5156 catalog_manager.cc:4841] TS 2f6324b1eacd45fea800a90cec8843c3 (127.3.246.65:42747): tablet 5f1864ef4b614db0beb29866c7dd7e9b (table test-table [id=cd678e7b72c74229b2e28aaec52ce163]) successfully deleted
Sep 24 07:18:28 dist-test-slave-rc1w krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.246.65@KRBTEST.COM
I20260924 07:18:28.384006 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:28.369386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.3.246.65:42747)
0924 07:18:28.369430 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:28.369432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:28.373874 (+ 4442us) server_negotiation.cc:407] Connection header received
0924 07:18:28.373993 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:28.373994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:28.374006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:28.374038 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:28.374783 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.375555 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.376633 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.376945 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.376987 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:28.380574 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:28.380583 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:28.380583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:28.380591 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:28.382628 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:28.383056 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:28.383059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:28.383060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:28.383127 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:28.383363 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:28.383364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:28.383364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:28.383539 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:28.383689 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:28.383791 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 07:18:28.383873 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 07:18:28.384253 5369 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46006
I20260924 07:18:28.392891 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:28.385327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.3.246.126:46425)
0924 07:18:28.385373 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:28.385375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:28.385501 (+ 126us) server_negotiation.cc:407] Connection header received
0924 07:18:28.385550 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:28.385551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:28.385563 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:28.385626 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:28.386250 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.386985 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.388021 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:28.388317 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:28.388395 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:28.389818 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:28.389820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:28.389820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:28.389828 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:28.391559 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:28.391968 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:28.391969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:28.391970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:28.392082 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:28.392323 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:28.392324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:28.392324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:28.392502 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:28.392605 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:28.392724 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 07:18:28.392800 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 07:18:28.393370 5172 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47490
I20260924 07:18:28.394217 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 5244
I20260924 07:18:28.401120 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 5384
I20260924 07:18:28.406392 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 5524
I20260924 07:18:28.411224 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 5140
2026-09-24T07:18:28Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2752 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:28 dist-test-slave-rc1w krb5kdc[5754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:28 dist-test-slave-rc1w krb5kdc[5754](info): set up 2 sockets
Sep 24 07:18:28 dist-test-slave-rc1w krb5kdc[5754](info): commencing operation
krb5kdc: starting...
W20260924 07:18:29.449512 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:29 dist-test-slave-rc1w krb5kdc[5754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:18:29.578393 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:36791
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:36791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:29.659147 5767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:29.659375 5767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:29.659410 5767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:29.660545 5767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:29.660614 5767 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:18:29.660637 5767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:29.660660 5767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:29.660691 5767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:29.660713 5767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:29.662202 5767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:36791
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:36791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:29.662478 5767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:29.662714 5767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:29.667568 5773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:29.667716 5767 server_base.cc:1034] running on GCE node
W20260924 07:18:29.667577 5772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:29.667618 5775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:29.668026 5767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:29.668311 5767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:29.669456 5767 hybrid_clock.cc:648] HybridClock initialized: now 1790234309669431 us; error 41 us; skew 500 ppm
Sep 24 07:18:29 dist-test-slave-rc1w krb5kdc[5754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:29.671988 5767 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:18:29.672399 5767 webserver.cc:460] Webserver started at http://127.3.246.126:35911/ using document root <none> and password file <none>
I20260924 07:18:29.672595 5767 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:29.672636 5767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:29.672725 5767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:29.674039 5767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "4d4ac125a02c454986f03c59d11853df"
format_stamp: "Formatted at 2026-09-24 07:18:29 on dist-test-slave-rc1w"
server_key: "8ce59b3dedc6aa345b787982845d9d36"
server_key_iv: "04a5afa0d596cf34e2bd41e33d9049d1"
server_key_version: "encryptionkey@0"
I20260924 07:18:29.674338 5767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "4d4ac125a02c454986f03c59d11853df"
format_stamp: "Formatted at 2026-09-24 07:18:29 on dist-test-slave-rc1w"
server_key: "8ce59b3dedc6aa345b787982845d9d36"
server_key_iv: "04a5afa0d596cf34e2bd41e33d9049d1"
server_key_version: "encryptionkey@0"
I20260924 07:18:29.675634 5767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:29.676415 5782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:29.676681 5767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:29.676800 5767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "4d4ac125a02c454986f03c59d11853df"
format_stamp: "Formatted at 2026-09-24 07:18:29 on dist-test-slave-rc1w"
server_key: "8ce59b3dedc6aa345b787982845d9d36"
server_key_iv: "04a5afa0d596cf34e2bd41e33d9049d1"
server_key_version: "encryptionkey@0"
I20260924 07:18:29.676880 5767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:29.690639 5767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:29.693001 5767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:29.693202 5767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:29.698228 5767 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:36791
I20260924 07:18:29.698434 5844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:36791 every 8 connection(s)
I20260924 07:18:29.698770 5767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:29.699599 5845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:29.702349 5845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df: Bootstrap starting.
I20260924 07:18:29.703126 5845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:29.703446 5845 log.cc:824] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:29.703460 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 5767
I20260924 07:18:29.703768 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:29.703970 4057 external_mini_cluster.cc:1428] Setting key a6cfb117c7ec801e715253a8ae77b71c
I20260924 07:18:29.704241 5845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df: No bootstrap required, opened a new log
I20260924 07:18:29.706651 5845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4ac125a02c454986f03c59d11853df" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 36791 } }
I20260924 07:18:29.706817 5845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:29.706866 5845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4ac125a02c454986f03c59d11853df, State: Initialized, Role: FOLLOWER
I20260924 07:18:29.707010 5845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4ac125a02c454986f03c59d11853df" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 36791 } }
I20260924 07:18:29.707111 5845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:29.707157 5845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:29.707211 5845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:29.707899 5845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4ac125a02c454986f03c59d11853df" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 36791 } }
I20260924 07:18:29.708009 5845 leader_election.cc:302] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d4ac125a02c454986f03c59d11853df; no voters:
I20260924 07:18:29.708202 5845 leader_election.cc:288] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:29.708264 5850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:29.708523 5850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [term 1 LEADER]: Becoming Leader. State: Replica: 4d4ac125a02c454986f03c59d11853df, State: Running, Role: LEADER
I20260924 07:18:29.708599 5845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:29.708772 5850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4ac125a02c454986f03c59d11853df" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 36791 } }
I20260924 07:18:29.709192 5850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d4ac125a02c454986f03c59d11853df. Latest consensus state: current_term: 1 leader_uuid: "4d4ac125a02c454986f03c59d11853df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4ac125a02c454986f03c59d11853df" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 36791 } } }
I20260924 07:18:29.709285 5850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:29.709327 5851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d4ac125a02c454986f03c59d11853df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4ac125a02c454986f03c59d11853df" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 36791 } } }
I20260924 07:18:29.709424 5851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df [sys.catalog]: This master's current role is: LEADER
W20260924 07:18:29.712383 5866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:18:29.712502 5866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:18:29.712612 5856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:29.712836 5856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 07:18:29 dist-test-slave-rc1w krb5kdc[5754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:29.714413 5856 catalog_manager.cc:1347] Generated new cluster ID: 6ba069b43bee44f6b0e133f39e224b0d
I20260924 07:18:29.714468 5856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:29.718604 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:29.704600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47700 (local address 127.3.246.126:36791)
0924 07:18:29.704743 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:29.704752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:29.704783 (+ 31us) server_negotiation.cc:407] Connection header received
0924 07:18:29.706655 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:29.706667 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:29.706730 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:29.706918 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:29.707748 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:29.709320 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:29.710530 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:29.711220 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:29.711371 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:29.714488 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:29.714492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:29.714493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:29.714506 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:29.716590 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:29.717081 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:29.717086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:29.717086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:29.717225 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:29.717445 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:29.717446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:29.717446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:29.717904 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:29.718070 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:29.718221 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 07:18:29.718373 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
I20260924 07:18:29.722260 5856 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:29.723038 5856 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:29.729815 5856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df: Generated new TSK 0
I20260924 07:18:29.730026 5856 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:18:29.762879 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:36791
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:29.845635 5871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:29.845883 5871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:29.845909 5871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:29.847049 5871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:29.847090 5871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:29.847111 5871 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:18:29.847124 5871 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:18:29.847141 5871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:29.848637 5871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.246.126:36791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:29.848910 5871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:29.849144 5871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:29.854144 5876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:29.854173 5877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:29.854275 5871 server_base.cc:1034] running on GCE node
W20260924 07:18:29.854153 5879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:29.854525 5871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:29.854732 5871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:29.855887 5871 hybrid_clock.cc:648] HybridClock initialized: now 1790234309855865 us; error 35 us; skew 500 ppm
Sep 24 07:18:29 dist-test-slave-rc1w krb5kdc[5754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:29.858419 5871 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:18:29.858860 5871 webserver.cc:460] Webserver started at http://127.3.246.65:35309/ using document root <none> and password file <none>
I20260924 07:18:29.859083 5871 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:29.859127 5871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:29.859270 5871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:29.860314 5871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "3b3909ef3d5d421aacb4364e76f4f2cc"
format_stamp: "Formatted at 2026-09-24 07:18:29 on dist-test-slave-rc1w"
server_key: "e5a86e576fd645631446566b5e86c1c9"
server_key_iv: "1765b5bfc67d37a4bd417ae5ef4b1418"
server_key_version: "encryptionkey@0"
I20260924 07:18:29.860667 5871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "3b3909ef3d5d421aacb4364e76f4f2cc"
format_stamp: "Formatted at 2026-09-24 07:18:29 on dist-test-slave-rc1w"
server_key: "e5a86e576fd645631446566b5e86c1c9"
server_key_iv: "1765b5bfc67d37a4bd417ae5ef4b1418"
server_key_version: "encryptionkey@0"
I20260924 07:18:29.862125 5871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:29.862913 5886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:29.863222 5871 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:29.863309 5871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "3b3909ef3d5d421aacb4364e76f4f2cc"
format_stamp: "Formatted at 2026-09-24 07:18:29 on dist-test-slave-rc1w"
server_key: "e5a86e576fd645631446566b5e86c1c9"
server_key_iv: "1765b5bfc67d37a4bd417ae5ef4b1418"
server_key_version: "encryptionkey@0"
I20260924 07:18:29.863359 5871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:29.883606 5871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:29.886402 5871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:29.886528 5871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:29.887077 5871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:29.887117 5871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:29.887141 5871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:29.887154 5871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:29.894438 5871 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:40821
Sep 24 07:18:29 dist-test-slave-rc1w krb5kdc[5754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:29.894520 6001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:40821 every 8 connection(s)
I20260924 07:18:29.894862 5871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:29.898546 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:29.887684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:36791 (local address 127.3.246.65:34429)
0924 07:18:29.887871 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:29.887911 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:29.888596 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:29.888914 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:29.888918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:29.889299 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:29.889591 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:29.889597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:29.890713 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:29.890714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:29.891603 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:29.891605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:29.892188 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:29.892190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:29.892363 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:29.893056 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:29.893066 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:29.894990 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:29.896932 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:29.896940 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:29.896942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:29.897174 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:29.897464 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:29.897465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:29.897465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:29.897576 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:29.897991 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:29.897997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:29.898144 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 07:18:29.898242 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 07:18:29.898374 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":122,"threads_started":1}
I20260924 07:18:29.898545 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 5871
I20260924 07:18:29.898653 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:29.887710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:34429 (local address 127.3.246.126:36791)
0924 07:18:29.887786 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:29.887789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:29.888018 (+ 229us) server_negotiation.cc:407] Connection header received
0924 07:18:29.888723 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:29.888724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:29.888743 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:29.888829 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:29.889728 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:29.890606 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:29.891717 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:29.892058 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:29.892153 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:29.895094 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:29.895095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:29.895096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:29.895103 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:29.896756 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:29.897272 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:29.897273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:29.897273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:29.897340 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:29.897666 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:29.897667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:29.897668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:29.897883 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:29.897951 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:29.898425 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 07:18:29.898531 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:18:29.898756 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:29.898965 4057 external_mini_cluster.cc:1428] Setting key cf82447d45fc6f493e6c7c4174acebe3
I20260924 07:18:29.899412 6002 heartbeater.cc:347] Connected to a master server at 127.3.246.126:36791
I20260924 07:18:29.899499 6002 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:29.899731 6002 heartbeater.cc:511] Master 127.3.246.126:36791 requested a full tablet report, sending...
I20260924 07:18:29.900277 5798 ts_manager.cc:194] Registered new tserver with Master: 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821)
I20260924 07:18:29.901242 5798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:34429
I20260924 07:18:29.907680 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:29.900499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:36791 (local address 127.3.246.65:50053)
0924 07:18:29.900721 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:29.900733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:29.900795 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:29.900923 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:29.900923 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:29.901111 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:29.901182 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:29.901184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:29.902142 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:29.902143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:29.902623 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:29.902624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:29.903166 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:29.903167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:29.903397 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:29.903944 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:29.903949 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:29.904288 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:29.906169 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:29.906170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:29.906170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:29.906788 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:29.907012 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:29.907012 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:29.907012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:29.907073 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:29.907389 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:29.907390 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:29.907445 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 07:18:29.907513 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:18:29.907602 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":146,"threads_started":1}
I20260924 07:18:29.907773 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:29.900540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:50053 (local address 127.3.246.126:36791)
0924 07:18:29.900591 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:29.900593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:29.900803 (+ 210us) server_negotiation.cc:407] Connection header received
0924 07:18:29.900853 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:29.900853 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:29.900864 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:29.900922 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:29.901257 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:29.902067 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:29.902711 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:29.903037 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:29.903112 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:29.904383 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:29.904385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:29.904385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:29.904394 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:29.906057 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:29.906876 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:29.906877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:29.906878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:29.906938 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:29.907141 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:29.907142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:29.907142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:29.907286 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:29.907369 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:29.907565 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 07:18:29.907662 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:18:29.930064 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:36791
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:30.013167 6011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:30.013370 6011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:30.013397 6011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:30.014555 6011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:30.014600 6011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:30.014623 6011 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:18:30.014637 6011 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:18:30.014659 6011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:30.016045 6011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.246.126:36791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:30.016228 6011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:30.016404 6011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:30.021320 6019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:30.021328 6016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:30.021593 6011 server_base.cc:1034] running on GCE node
W20260924 07:18:30.021325 6017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:30.021948 6011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:30.022154 6011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:30.023303 6011 hybrid_clock.cc:648] HybridClock initialized: now 1790234310023289 us; error 29 us; skew 500 ppm
Sep 24 07:18:30 dist-test-slave-rc1w krb5kdc[5754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:30.025822 6011 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:18:30.026264 6011 webserver.cc:460] Webserver started at http://127.3.246.66:43993/ using document root <none> and password file <none>
I20260924 07:18:30.026482 6011 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:30.026526 6011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:30.026664 6011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:30.027680 6011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "babe220450db4a5691db9ab7de4574cb"
format_stamp: "Formatted at 2026-09-24 07:18:30 on dist-test-slave-rc1w"
server_key: "0e2a41ccfe351bc0211b6600342fa8b7"
server_key_iv: "3ba207a9bb583ea528c1bb1d59747426"
server_key_version: "encryptionkey@0"
I20260924 07:18:30.028045 6011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "babe220450db4a5691db9ab7de4574cb"
format_stamp: "Formatted at 2026-09-24 07:18:30 on dist-test-slave-rc1w"
server_key: "0e2a41ccfe351bc0211b6600342fa8b7"
server_key_iv: "3ba207a9bb583ea528c1bb1d59747426"
server_key_version: "encryptionkey@0"
I20260924 07:18:30.029381 6011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:30.030155 6026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:30.030326 6011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:30.030423 6011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "babe220450db4a5691db9ab7de4574cb"
format_stamp: "Formatted at 2026-09-24 07:18:30 on dist-test-slave-rc1w"
server_key: "0e2a41ccfe351bc0211b6600342fa8b7"
server_key_iv: "3ba207a9bb583ea528c1bb1d59747426"
server_key_version: "encryptionkey@0"
I20260924 07:18:30.030505 6011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:30.038388 6011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:30.040210 6011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:30.040367 6011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:30.040830 6011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:30.040900 6011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:30.040966 6011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:30.040997 6011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:18:30 dist-test-slave-rc1w krb5kdc[5754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:30.048194 6011 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:35021
I20260924 07:18:30.048266 6141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:35021 every 8 connection(s)
I20260924 07:18:30.048542 6011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:30.051525 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.041408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:39649 (local address 127.3.246.126:36791)
0924 07:18:30.041485 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.041488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.041635 (+ 147us) server_negotiation.cc:407] Connection header received
0924 07:18:30.042181 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.042183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.042197 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.042277 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:30.042995 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.043851 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.045517 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.045873 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.045953 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.048601 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:30.048603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:30.048603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:30.048611 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:30.050144 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:30.050582 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:30.050583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:30.050584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:30.050642 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:30.050894 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:30.050895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:30.050896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:30.051009 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:30.051083 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.051346 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.051431 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:18:30.051739 6035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.041349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:36791 (local address 127.3.246.66:39649)
0924 07:18:30.041510 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:30.041547 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:30.042072 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:30.042362 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:30.042366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:30.042698 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:30.042887 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.042891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.043962 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.043963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.045425 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.045428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.045971 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.045972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.046192 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.046638 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:30.046651 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:30.048498 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:30.050247 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:30.050254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:30.050256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:30.050491 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:30.050713 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:30.050713 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:30.050714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:30.050823 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:30.051076 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:30.051082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:30.051215 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 07:18:30.051324 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 07:18:30.051489 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260924 07:18:30.052237 6142 heartbeater.cc:347] Connected to a master server at 127.3.246.126:36791
I20260924 07:18:30.052351 6142 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:30.052531 6142 heartbeater.cc:511] Master 127.3.246.126:36791 requested a full tablet report, sending...
I20260924 07:18:30.052874 5798 ts_manager.cc:194] Registered new tserver with Master: babe220450db4a5691db9ab7de4574cb (127.3.246.66:35021)
I20260924 07:18:30.053319 5798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:39649
I20260924 07:18:30.054850 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 6011
I20260924 07:18:30.054960 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:30.055140 4057 external_mini_cluster.cc:1428] Setting key 24006be6d41f31ea0b314c2a1e05829d
I20260924 07:18:30.059520 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.053348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:33785 (local address 127.3.246.126:36791)
0924 07:18:30.053391 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.053393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.053482 (+ 89us) server_negotiation.cc:407] Connection header received
0924 07:18:30.053553 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.053553 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.053563 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.053610 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:30.054327 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.054809 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.055761 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.055956 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.056031 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.056905 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:30.056906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:30.056907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:30.056914 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:30.058310 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:30.058691 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:30.058692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:30.058693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:30.058751 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:30.058953 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:30.058954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:30.058955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:30.059102 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:30.059179 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.059351 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.059429 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:18:30.059525 6148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.053259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:36791 (local address 127.3.246.66:33785)
0924 07:18:30.053419 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:30.053431 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:30.053494 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:30.054034 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:30.054034 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:30.054179 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:30.054228 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.054230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.054905 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.054906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.055682 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.055684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.056042 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.056042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.056117 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.056548 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:30.056552 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:30.056805 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:30.058429 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:30.058430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:30.058430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:30.058603 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:30.058819 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:30.058820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:30.058820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:30.058879 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:30.059186 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:30.059187 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:30.059232 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 07:18:30.059324 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 07:18:30.059433 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:18:30.081862 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:36791
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:30.164947 6151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:30.165148 6151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:30.165171 6151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:30.166491 6151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:30.166530 6151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:30.166553 6151 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:18:30.166584 6151 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:18:30.166617 6151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:30.168098 6151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.246.126:36791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:30.168329 6151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:30.168610 6151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:30.173882 6157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:30.173913 6159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:30.174073 6156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:30.174683 6151 server_base.cc:1034] running on GCE node
I20260924 07:18:30.174912 6151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:30.175194 6151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:30.176405 6151 hybrid_clock.cc:648] HybridClock initialized: now 1790234310176382 us; error 44 us; skew 500 ppm
Sep 24 07:18:30 dist-test-slave-rc1w krb5kdc[5754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:30.180104 6151 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:18:30.180816 6151 webserver.cc:460] Webserver started at http://127.3.246.67:39145/ using document root <none> and password file <none>
I20260924 07:18:30.181098 6151 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:30.181143 6151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:30.181246 6151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:30.182595 6151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "c9158526bb6744c89df80d37e38a5579"
format_stamp: "Formatted at 2026-09-24 07:18:30 on dist-test-slave-rc1w"
server_key: "ff6da1f8ec7b08c9f0c2e348d6f995bc"
server_key_iv: "e65e7434fedae9066621a8e30e9fdf24"
server_key_version: "encryptionkey@0"
I20260924 07:18:30.183063 6151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "c9158526bb6744c89df80d37e38a5579"
format_stamp: "Formatted at 2026-09-24 07:18:30 on dist-test-slave-rc1w"
server_key: "ff6da1f8ec7b08c9f0c2e348d6f995bc"
server_key_iv: "e65e7434fedae9066621a8e30e9fdf24"
server_key_version: "encryptionkey@0"
I20260924 07:18:30.184777 6151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:18:30.185611 6166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:30.185832 6151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:30.185935 6151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "c9158526bb6744c89df80d37e38a5579"
format_stamp: "Formatted at 2026-09-24 07:18:30 on dist-test-slave-rc1w"
server_key: "ff6da1f8ec7b08c9f0c2e348d6f995bc"
server_key_iv: "e65e7434fedae9066621a8e30e9fdf24"
server_key_version: "encryptionkey@0"
I20260924 07:18:30.186009 6151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:30.206542 6151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:30.209061 6151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:30.209218 6151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:30.209913 6151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:30.209960 6151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:30.209985 6151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:30.209997 6151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:30.216648 6151 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:38227
I20260924 07:18:30.216723 6281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:38227 every 8 connection(s)
I20260924 07:18:30.217022 6151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:18:30 dist-test-slave-rc1w krb5kdc[5754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:30.217456 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 6151
I20260924 07:18:30.217582 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:30.217842 4057 external_mini_cluster.cc:1428] Setting key d5478bd2c65122e3dae8c962fcd3bf96
I20260924 07:18:30.221944 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.210405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:57549 (local address 127.3.246.126:36791)
0924 07:18:30.210462 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.210464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.210669 (+ 205us) server_negotiation.cc:407] Connection header received
0924 07:18:30.211532 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.211534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.211547 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.211595 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:30.212486 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.213314 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.214478 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.214809 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.214901 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.218553 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:30.218554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:30.218555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:30.218563 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:30.220234 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:30.220746 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:30.220747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:30.220748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:30.220830 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:30.221127 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:30.221128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:30.221128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:30.221281 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:30.221352 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.221741 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.221848 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:18:30.222199 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.210398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:36791 (local address 127.3.246.67:57549)
0924 07:18:30.210554 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:30.210594 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:30.211405 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:30.211689 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:30.211694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:30.212075 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:30.212396 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.212403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.213418 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.213419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.214372 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.214375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.214921 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.214923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.215098 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.215718 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:30.215729 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:30.218442 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:30.220328 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:30.220334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:30.220337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:30.220667 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:30.220904 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:30.220905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:30.220906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:30.221056 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:30.221362 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:30.221368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:30.221546 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 07:18:30.221727 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 07:18:30.221928 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260924 07:18:30.222681 6282 heartbeater.cc:347] Connected to a master server at 127.3.246.126:36791
I20260924 07:18:30.222813 6282 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:30.223040 6282 heartbeater.cc:511] Master 127.3.246.126:36791 requested a full tablet report, sending...
I20260924 07:18:30.223441 5798 ts_manager.cc:194] Registered new tserver with Master: c9158526bb6744c89df80d37e38a5579 (127.3.246.67:38227)
I20260924 07:18:30.223918 5798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:57549
I20260924 07:18:30.229417 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:18:30.230948 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.223756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:36791 (local address 127.3.246.67:44327)
0924 07:18:30.223940 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:30.223953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:30.224005 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:30.224245 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:30.224246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:30.224473 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:30.224545 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.224547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.225598 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.225599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.226498 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.226502 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.227074 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.227075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.227138 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.227592 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:30.227596 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:30.227911 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:30.229629 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:30.229629 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:30.229630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:30.229872 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:30.230199 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:30.230200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:30.230200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:30.230286 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:30.230575 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:30.230576 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:30.230632 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 07:18:30.230721 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 07:18:30.230832 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":121,"threads_started":1}
I20260924 07:18:30.230980 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.224088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:44327 (local address 127.3.246.126:36791)
0924 07:18:30.224126 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.224128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.224135 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:18:30.224154 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.224155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.224164 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.224218 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:30.224658 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.225478 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.226642 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.226992 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.227076 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.227996 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:30.227997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:30.227998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:30.228005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:30.229538 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:30.230058 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:30.230060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:30.230060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:30.230121 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:30.230352 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:30.230353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:30.230354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:30.230494 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:30.230565 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.230765 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.230846 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 07:18:30 dist-test-slave-rc1w krb5kdc[5754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:18:30 dist-test-slave-rc1w krb5kdc[5754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:30.252403 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.245222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47716 (local address 127.3.246.126:36791)
0924 07:18:30.245283 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.245286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.245300 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:18:30.245323 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.245324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.245333 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.245383 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:30.246107 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.246589 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.247362 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.247563 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.247627 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.249912 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:30.249913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:30.249913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:30.249920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:30.251370 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:30.251674 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:30.251675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:30.251675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:30.251723 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:30.251956 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:30.251957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:30.251957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:30.252057 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:30.252128 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.252275 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.252338 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:18:30.254056 5798 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:30.255656 5798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:30.262485 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.258046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:40821 (local address 127.0.0.1:39764)
0924 07:18:30.258760 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:30.258769 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:30.258813 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:30.259007 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:30.259007 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:30.259009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:30.259083 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.259085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.260036 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.260037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.261186 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.261189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.262173 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.262174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.262250 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.262264 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 07:18:30.262342 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:18:30.262408 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":79,"thread_start_us":74,"threads_started":1}
I20260924 07:18:30.262542 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.258274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.3.246.65:40821)
0924 07:18:30.258476 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.258482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.258869 (+ 387us) server_negotiation.cc:407] Connection header received
0924 07:18:30.258913 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.258916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.258931 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.259005 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:30.259155 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.259938 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.261287 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.262083 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.262165 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.262230 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.262368 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.262418 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260924 07:18:30.262807 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.258149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47624 (local address 127.3.246.67:38227)
0924 07:18:30.258500 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.258505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.258629 (+ 124us) server_negotiation.cc:407] Connection header received
0924 07:18:30.258734 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.258736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.258754 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.258830 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:30.259081 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.259874 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.261463 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.262186 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.262248 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.262293 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.262673 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.262716 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":76,"threads_started":1}
I20260924 07:18:30.262614 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.258045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:38227 (local address 127.0.0.1:47624)
0924 07:18:30.258509 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:30.258548 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:30.258651 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:30.258923 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:30.258926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:30.258928 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:30.258998 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.259000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.259963 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.259964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.261371 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.261374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.262352 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.262353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.262429 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.262434 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:18:30.262487 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:18:30.262530 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":41,"thread_start_us":58,"threads_started":1}
I20260924 07:18:30.263605 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.258320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.3.246.66:35021)
0924 07:18:30.258493 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.258497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.260032 (+ 1535us) server_negotiation.cc:407] Connection header received
0924 07:18:30.260077 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.260079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.260095 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.260170 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:30.260351 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.261428 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.262441 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.263269 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.263340 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.263391 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.263480 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.263514 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260924 07:18:30.263628 5936 tablet_service.cc:1467] Processing CreateTablet for tablet 14e7f0d9cf104ed194145994d1c1d2de (DEFAULT_TABLE table=test-table [id=c24cf0ac1cdf42289b17ef50dc395c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:30.263984 5936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e7f0d9cf104ed194145994d1c1d2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:30.263526 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.258065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:35021 (local address 127.0.0.1:57544)
0924 07:18:30.259082 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:30.259091 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:30.259143 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:30.260184 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:30.260185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:30.260187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:30.260265 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.260267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.261510 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.261510 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.262357 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.262359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.263357 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.263357 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.263408 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.263412 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:30.263455 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 07:18:30.263482 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":41,"thread_start_us":102,"threads_started":1}
I20260924 07:18:30.266405 6203 tablet_service.cc:1467] Processing CreateTablet for tablet 14e7f0d9cf104ed194145994d1c1d2de (DEFAULT_TABLE table=test-table [id=c24cf0ac1cdf42289b17ef50dc395c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:30.266387 6076 tablet_service.cc:1467] Processing CreateTablet for tablet 14e7f0d9cf104ed194145994d1c1d2de (DEFAULT_TABLE table=test-table [id=c24cf0ac1cdf42289b17ef50dc395c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:30.266752 6203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e7f0d9cf104ed194145994d1c1d2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:30.266758 6076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e7f0d9cf104ed194145994d1c1d2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:30.267390 6305 tablet_bootstrap.cc:492] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc: Bootstrap starting.
I20260924 07:18:30.268102 6305 tablet_bootstrap.cc:654] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:30.268489 6305 log.cc:824] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:30.269660 6305 tablet_bootstrap.cc:492] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc: No bootstrap required, opened a new log
I20260924 07:18:30.269829 6305 ts_tablet_manager.cc:1397] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:18:30.269853 6307 tablet_bootstrap.cc:492] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb: Bootstrap starting.
I20260924 07:18:30.270087 6308 tablet_bootstrap.cc:492] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579: Bootstrap starting.
I20260924 07:18:30.270642 6307 tablet_bootstrap.cc:654] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:30.270973 6307 log.cc:824] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:30.271004 6308 tablet_bootstrap.cc:654] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:30.271312 6308 log.cc:824] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:30.271756 6307 tablet_bootstrap.cc:492] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb: No bootstrap required, opened a new log
I20260924 07:18:30.271827 6307 ts_tablet_manager.cc:1397] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:30.271840 6305 raft_consensus.cc:348] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } }
I20260924 07:18:30.272029 6305 raft_consensus.cc:374] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:30.272046 6308 tablet_bootstrap.cc:492] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579: No bootstrap required, opened a new log
I20260924 07:18:30.272071 6305 raft_consensus.cc:729] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3909ef3d5d421aacb4364e76f4f2cc, State: Initialized, Role: FOLLOWER
I20260924 07:18:30.272102 6308 ts_tablet_manager.cc:1397] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:18:30.272203 6305 consensus_queue.cc:260] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } }
I20260924 07:18:30.272502 6002 heartbeater.cc:503] Master 127.3.246.126:36791 was elected leader, sending a full tablet report...
I20260924 07:18:30.272490 6305 ts_tablet_manager.cc:1428] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:18:30.274085 6308 raft_consensus.cc:348] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } }
I20260924 07:18:30.274255 6308 raft_consensus.cc:374] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:30.274290 6308 raft_consensus.cc:729] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9158526bb6744c89df80d37e38a5579, State: Initialized, Role: FOLLOWER
I20260924 07:18:30.274207 6307 raft_consensus.cc:348] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } }
I20260924 07:18:30.274336 6307 raft_consensus.cc:374] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:30.274387 6307 raft_consensus.cc:729] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babe220450db4a5691db9ab7de4574cb, State: Initialized, Role: FOLLOWER
I20260924 07:18:30.274410 6308 consensus_queue.cc:260] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } }
I20260924 07:18:30.274488 6307 consensus_queue.cc:260] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } }
I20260924 07:18:30.274674 6308 ts_tablet_manager.cc:1428] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:30.274726 6307 ts_tablet_manager.cc:1428] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:30.274734 6282 heartbeater.cc:503] Master 127.3.246.126:36791 was elected leader, sending a full tablet report...
I20260924 07:18:30.274783 6142 heartbeater.cc:503] Master 127.3.246.126:36791 was elected leader, sending a full tablet report...
W20260924 07:18:30.299170 6143 tablet.cc:2369] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:18:30.395746 6003 tablet.cc:2369] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:18:30.467675 6283 tablet.cc:2369] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:18:30.635617 6313 raft_consensus.cc:482] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:30.635751 6313 raft_consensus.cc:504] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } }
I20260924 07:18:30.636204 6313 leader_election.cc:288] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821), c9158526bb6744c89df80d37e38a5579 (127.3.246.67:38227)
I20260924 07:18:30.639413 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.636610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:60257 (local address 127.3.246.65:40821)
0924 07:18:30.636671 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.636673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.636778 (+ 105us) server_negotiation.cc:407] Connection header received
0924 07:18:30.636861 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.636862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.636871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.636952 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:30.637130 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.637617 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.638667 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.639029 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.639127 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.639136 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.639296 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.639336 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:18:30.639430 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.636496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:40821 (local address 127.3.246.66:60257)
0924 07:18:30.636698 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:30.636709 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:30.636794 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:30.636957 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:30.636958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:30.636964 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:30.637049 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.637051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.637740 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.637741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.638562 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.638565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.639127 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.639128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.639203 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.639208 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:18:30.639289 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:18:30.639330 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":23,"thread_start_us":70,"threads_started":1}
I20260924 07:18:30.639516 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.636542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:43325 (local address 127.3.246.67:38227)
0924 07:18:30.636593 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.636595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.636754 (+ 159us) server_negotiation.cc:407] Connection header received
0924 07:18:30.636835 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.636836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.636847 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.636946 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:30.637102 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.637608 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.638729 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.639098 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.639210 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.639219 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.639393 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.639438 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:18:30.639688 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.636439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:38227 (local address 127.3.246.66:43325)
0924 07:18:30.636656 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:30.636667 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:30.636772 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:30.636956 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:30.636957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:30.636960 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:30.637028 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.637030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.637721 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.637722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.638596 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.638599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.639216 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.639217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.639297 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.639301 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:30.639372 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:18:30.639418 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":88,"thread_start_us":97,"threads_started":1}
I20260924 07:18:30.639801 5956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e7f0d9cf104ed194145994d1c1d2de" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" is_pre_election: true
I20260924 07:18:30.639989 5956 raft_consensus.cc:2454] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 0.
I20260924 07:18:30.640100 6236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e7f0d9cf104ed194145994d1c1d2de" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9158526bb6744c89df80d37e38a5579" is_pre_election: true
I20260924 07:18:30.640267 6236 raft_consensus.cc:2454] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 0.
I20260924 07:18:30.640246 6027 leader_election.cc:302] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b3909ef3d5d421aacb4364e76f4f2cc, babe220450db4a5691db9ab7de4574cb; no voters:
I20260924 07:18:30.640383 6313 raft_consensus.cc:2781] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:30.640442 6313 raft_consensus.cc:482] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:30.640465 6313 raft_consensus.cc:3037] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:30.641158 6313 raft_consensus.cc:504] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } }
I20260924 07:18:30.641306 6313 leader_election.cc:288] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 election: Requested vote from peers 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821), c9158526bb6744c89df80d37e38a5579 (127.3.246.67:38227)
I20260924 07:18:30.641484 5956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e7f0d9cf104ed194145994d1c1d2de" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc"
I20260924 07:18:30.641568 5956 raft_consensus.cc:3037] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:30.641630 6236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e7f0d9cf104ed194145994d1c1d2de" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9158526bb6744c89df80d37e38a5579"
I20260924 07:18:30.641763 6236 raft_consensus.cc:3037] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:30.642378 5956 raft_consensus.cc:2454] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 1.
I20260924 07:18:30.642550 6027 leader_election.cc:302] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b3909ef3d5d421aacb4364e76f4f2cc, babe220450db4a5691db9ab7de4574cb; no voters:
I20260924 07:18:30.642638 6313 raft_consensus.cc:2781] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:30.642637 6236 raft_consensus.cc:2454] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 1.
I20260924 07:18:30.642792 6313 raft_consensus.cc:686] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [term 1 LEADER]: Becoming Leader. State: Replica: babe220450db4a5691db9ab7de4574cb, State: Running, Role: LEADER
I20260924 07:18:30.642891 6313 consensus_queue.cc:237] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } }
I20260924 07:18:30.643783 5798 catalog_manager.cc:5495] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb reported cstate change: term changed from 0 to 1, leader changed from <none> to babe220450db4a5691db9ab7de4574cb (127.3.246.66). New cstate: current_term: 1 leader_uuid: "babe220450db4a5691db9ab7de4574cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } health_report { overall_health: HEALTHY } } }
Sep 24 07:18:30 dist-test-slave-rc1w krb5kdc[5754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:30.722062 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.714441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:36791 (local address 127.0.0.1:47728)
0924 07:18:30.714600 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:30.714612 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:30.714693 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:30.714902 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:30.714903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:30.715334 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:30.715394 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.715417 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.716178 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.716179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.716833 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.716835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.717216 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.717218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.717271 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.717823 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:30.717829 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:30.719253 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:30.720763 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:30.720770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:30.720773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:30.720983 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:30.721211 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:30.721211 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:30.721212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:30.721288 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:30.721562 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:30.721569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:30.721618 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 07:18:30.721761 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 07:18:30.721925 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20260924 07:18:30.722067 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.714571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.3.246.126:36791)
0924 07:18:30.714612 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.714615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.714680 (+ 65us) server_negotiation.cc:407] Connection header received
0924 07:18:30.714756 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.714757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.714768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.714857 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:30.715505 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.716103 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.716924 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.717126 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.717210 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.719365 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:30.719367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:30.719367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:30.719375 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:30.720639 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:30.721068 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:30.721070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:30.721070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:30.721131 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:30.721356 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:30.721357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:30.721358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:30.721486 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:30.721571 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.721791 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.721981 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":17}
I20260924 07:18:30.724380 5798 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:47728:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:18:30.724601 5798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:30.726233 6203 tablet_service.cc:1467] Processing CreateTablet for tablet 91e24e3b13cd4511b44c51307c1431e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac4ae9e159dd4234a9d1efc7d58bfd4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:18:30.726224 5936 tablet_service.cc:1467] Processing CreateTablet for tablet 91e24e3b13cd4511b44c51307c1431e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac4ae9e159dd4234a9d1efc7d58bfd4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:18:30.726253 6076 tablet_service.cc:1467] Processing CreateTablet for tablet 91e24e3b13cd4511b44c51307c1431e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac4ae9e159dd4234a9d1efc7d58bfd4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:18:30.726402 6076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e24e3b13cd4511b44c51307c1431e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:30.726402 5936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e24e3b13cd4511b44c51307c1431e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:30.726423 6203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e24e3b13cd4511b44c51307c1431e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:30.727943 6307 tablet_bootstrap.cc:492] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb: Bootstrap starting.
I20260924 07:18:30.727948 6305 tablet_bootstrap.cc:492] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc: Bootstrap starting.
I20260924 07:18:30.727984 6308 tablet_bootstrap.cc:492] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579: Bootstrap starting.
I20260924 07:18:30.728348 6305 tablet_bootstrap.cc:654] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:30.728385 6307 tablet_bootstrap.cc:654] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:30.728394 6308 tablet_bootstrap.cc:654] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:30.729009 6308 tablet_bootstrap.cc:492] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579: No bootstrap required, opened a new log
I20260924 07:18:30.729027 6305 tablet_bootstrap.cc:492] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc: No bootstrap required, opened a new log
I20260924 07:18:30.729077 6308 ts_tablet_manager.cc:1397] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:30.729071 6305 ts_tablet_manager.cc:1397] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:30.729245 6305 raft_consensus.cc:348] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.729307 6305 raft_consensus.cc:374] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:30.729321 6305 raft_consensus.cc:729] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3909ef3d5d421aacb4364e76f4f2cc, State: Initialized, Role: FOLLOWER
I20260924 07:18:30.729378 6305 consensus_queue.cc:260] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.729358 6308 raft_consensus.cc:348] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.729432 6308 raft_consensus.cc:374] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:30.729465 6308 raft_consensus.cc:729] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9158526bb6744c89df80d37e38a5579, State: Initialized, Role: FOLLOWER
I20260924 07:18:30.729485 6305 ts_tablet_manager.cc:1428] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:30.729513 6308 consensus_queue.cc:260] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.729545 6311 tablet_replica.cc:439] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } }
I20260924 07:18:30.729604 6308 ts_tablet_manager.cc:1428] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:30.729632 6311 tablet_replica.cc:442] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:30.729743 6307 tablet_bootstrap.cc:492] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb: No bootstrap required, opened a new log
I20260924 07:18:30.729689 6312 tablet_replica.cc:439] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } }
I20260924 07:18:30.729782 6307 ts_tablet_manager.cc:1397] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:30.729782 6312 tablet_replica.cc:442] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:30.730134 6307 raft_consensus.cc:348] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.730201 6307 raft_consensus.cc:374] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:30.730216 6307 raft_consensus.cc:729] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babe220450db4a5691db9ab7de4574cb, State: Initialized, Role: FOLLOWER
I20260924 07:18:30.730278 6307 consensus_queue.cc:260] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.730366 6307 ts_tablet_manager.cc:1428] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:30.730461 6313 tablet_replica.cc:439] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } }
I20260924 07:18:30.730527 6313 tablet_replica.cc:442] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:30.747648 6313 raft_consensus.cc:482] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:30.747753 6313 raft_consensus.cc:504] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.747943 6313 leader_election.cc:288] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821), c9158526bb6744c89df80d37e38a5579 (127.3.246.67:38227)
I20260924 07:18:30.748152 6236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e24e3b13cd4511b44c51307c1431e4" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9158526bb6744c89df80d37e38a5579" is_pre_election: true
I20260924 07:18:30.748173 5956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e24e3b13cd4511b44c51307c1431e4" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" is_pre_election: true
I20260924 07:18:30.748272 6236 raft_consensus.cc:2454] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 0.
I20260924 07:18:30.748293 5956 raft_consensus.cc:2454] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 0.
I20260924 07:18:30.748482 6029 leader_election.cc:302] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: babe220450db4a5691db9ab7de4574cb, c9158526bb6744c89df80d37e38a5579; no voters:
I20260924 07:18:30.748603 6313 raft_consensus.cc:2781] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:30.748657 6313 raft_consensus.cc:482] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:30.748689 6313 raft_consensus.cc:3037] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:30.749372 6313 raft_consensus.cc:504] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.749512 6313 leader_election.cc:288] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 election: Requested vote from peers 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821), c9158526bb6744c89df80d37e38a5579 (127.3.246.67:38227)
I20260924 07:18:30.749748 5956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e24e3b13cd4511b44c51307c1431e4" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc"
I20260924 07:18:30.749752 6236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e24e3b13cd4511b44c51307c1431e4" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9158526bb6744c89df80d37e38a5579"
I20260924 07:18:30.749826 6236 raft_consensus.cc:3037] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:30.749837 5956 raft_consensus.cc:3037] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:30.750413 5956 raft_consensus.cc:2454] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 1.
I20260924 07:18:30.750469 6236 raft_consensus.cc:2454] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 1.
I20260924 07:18:30.750607 6027 leader_election.cc:302] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b3909ef3d5d421aacb4364e76f4f2cc, babe220450db4a5691db9ab7de4574cb; no voters:
I20260924 07:18:30.750747 6313 raft_consensus.cc:2781] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:30.750834 6313 raft_consensus.cc:686] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [term 1 LEADER]: Becoming Leader. State: Replica: babe220450db4a5691db9ab7de4574cb, State: Running, Role: LEADER
I20260924 07:18:30.750886 6313 consensus_queue.cc:237] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.751242 6316 tablet_replica.cc:439] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb: TxnStatusTablet state changed. Reason: New leader babe220450db4a5691db9ab7de4574cb. Latest consensus state: current_term: 1 leader_uuid: "babe220450db4a5691db9ab7de4574cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } }
I20260924 07:18:30.751361 6316 tablet_replica.cc:442] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:18:30.751575 6328 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:18:30.751535 5799 catalog_manager.cc:5495] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb reported cstate change: term changed from 0 to 1, leader changed from <none> to babe220450db4a5691db9ab7de4574cb (127.3.246.66). New cstate: current_term: 1 leader_uuid: "babe220450db4a5691db9ab7de4574cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } health_report { overall_health: UNKNOWN } } }
I20260924 07:18:30.751700 6328 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:18:30.759332 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.756738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.3.246.66:35021)
0924 07:18:30.756804 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.756806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.756935 (+ 129us) server_negotiation.cc:407] Connection header received
0924 07:18:30.756971 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.756972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.756991 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.757054 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:30.757224 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.757781 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.758546 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.758754 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.758829 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.759018 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:30.759113 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:30.759212 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.759242 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.759263 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:18:30.759356 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.756681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:35021 (local address 127.0.0.1:57554)
0924 07:18:30.756809 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:30.756822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:30.756884 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:30.757107 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:30.757108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:30.757111 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:18:30.757164 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.757167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.757863 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.757864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.758444 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.758446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.758828 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.758829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.758889 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.758903 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:18:30.759182 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:18:30.759192 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 07:18:30.759228 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 07:18:30.759283 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 07:18:30.761269 6236 raft_consensus.cc:1261] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Refusing update from remote peer babe220450db4a5691db9ab7de4574cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:30.761346 5956 raft_consensus.cc:1261] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Refusing update from remote peer babe220450db4a5691db9ab7de4574cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:30.761579 6316 consensus_queue.cc:1035] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:30.761701 6313 consensus_queue.cc:1035] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:30.762837 6312 tablet_replica.cc:439] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579: TxnStatusTablet state changed. Reason: New leader babe220450db4a5691db9ab7de4574cb. Latest consensus state: current_term: 1 leader_uuid: "babe220450db4a5691db9ab7de4574cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } }
I20260924 07:18:30.762846 6311 tablet_replica.cc:439] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc: TxnStatusTablet state changed. Reason: New leader babe220450db4a5691db9ab7de4574cb. Latest consensus state: current_term: 1 leader_uuid: "babe220450db4a5691db9ab7de4574cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } }
I20260924 07:18:30.762933 6312 tablet_replica.cc:442] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:30.762938 6311 tablet_replica.cc:442] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:30.764357 6316 tablet_replica.cc:439] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "babe220450db4a5691db9ab7de4574cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } }
I20260924 07:18:30.764442 6316 tablet_replica.cc:442] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:18:30.764540 6316 tablet_replica.cc:439] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "babe220450db4a5691db9ab7de4574cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } }
I20260924 07:18:30.764590 6316 tablet_replica.cc:442] T 91e24e3b13cd4511b44c51307c1431e4 P babe220450db4a5691db9ab7de4574cb: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:18:30.764633 6311 tablet_replica.cc:439] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "babe220450db4a5691db9ab7de4574cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } }
I20260924 07:18:30.764707 6311 tablet_replica.cc:442] T 91e24e3b13cd4511b44c51307c1431e4 P 3b3909ef3d5d421aacb4364e76f4f2cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:30.765034 6333 mvcc.cc:205] Tried to move back new op lower bound from 7332799736875003904 to 7332799736836030464. Current Snapshot: MvccSnapshot[applied={T|T < 7332799736875003904}]
I20260924 07:18:30.765074 6312 tablet_replica.cc:439] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "babe220450db4a5691db9ab7de4574cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } } }
I20260924 07:18:30.765163 6312 tablet_replica.cc:442] T 91e24e3b13cd4511b44c51307c1431e4 P c9158526bb6744c89df80d37e38a5579: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:30.765729 6332 mvcc.cc:205] Tried to move back new op lower bound from 7332799736875003904 to 7332799736836030464. Current Snapshot: MvccSnapshot[applied={T|T < 7332799736875003904}]
I20260924 07:18:30.774255 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.771538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.3.246.66:35021)
0924 07:18:30.771583 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.771585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.771718 (+ 133us) server_negotiation.cc:407] Connection header received
0924 07:18:30.771777 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.771778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.771789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.771843 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:30.771989 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.772733 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.773504 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.773826 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.773890 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.774034 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:30.774068 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:30.774128 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.774162 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.774182 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:18:30.779620 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.776704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:40745 (local address 127.3.246.66:35021)
0924 07:18:30.776765 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:30.776767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:30.776834 (+ 67us) server_negotiation.cc:407] Connection header received
0924 07:18:30.776880 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:30.776881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:30.776890 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:30.776944 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:30.777063 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.777815 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.778855 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.779148 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.779219 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.779368 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:30.779400 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:30.779453 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:30.779500 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 07:18:30.779517 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:18:30.779626 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:30.776641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:35021 (local address 127.3.246.66:40745)
0924 07:18:30.776783 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:30.776792 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:30.776841 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:30.776953 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:30.776953 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:30.776954 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:18:30.777001 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.777002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.777883 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.777884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.778772 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:30.778774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:30.779220 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:30.779221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:30.779269 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:30.779281 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:18:30.779461 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:18:30.779462 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 07:18:30.779497 (+ 35us) client_negotiation.cc:240] Negotiation successful
0924 07:18:30.779533 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260924 07:18:30.782032 5956 raft_consensus.cc:1261] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Refusing update from remote peer babe220450db4a5691db9ab7de4574cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:30.782060 6236 raft_consensus.cc:1261] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Refusing update from remote peer babe220450db4a5691db9ab7de4574cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:30.782277 6313 consensus_queue.cc:1035] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:30.782356 6316 consensus_queue.cc:1035] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:30.783517 6331 mvcc.cc:205] Tried to move back new op lower bound from 7332799736961703936 to 7332799736393781248. Current Snapshot: MvccSnapshot[applied={T|T < 7332799736961703936}]
I20260924 07:18:30.797003 5799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:30.797134 5799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:30.797935 5799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df: Sending DeleteTablet for 3 replicas of tablet 14e7f0d9cf104ed194145994d1c1d2de
I20260924 07:18:30.798259 6076 tablet_service.cc:1514] Processing DeleteTablet for tablet 14e7f0d9cf104ed194145994d1c1d2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:30 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:57544
I20260924 07:18:30.798336 5936 tablet_service.cc:1514] Processing DeleteTablet for tablet 14e7f0d9cf104ed194145994d1c1d2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:30 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:39764
I20260924 07:18:30.798388 6203 tablet_service.cc:1514] Processing DeleteTablet for tablet 14e7f0d9cf104ed194145994d1c1d2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:30 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:47624
I20260924 07:18:30.798543 6349 tablet_replica.cc:330] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb: stopping tablet replica
I20260924 07:18:30.798664 6349 raft_consensus.cc:2229] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:30.798767 6349 raft_consensus.cc:2258] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:30.798869 5799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:30.799014 5799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:30.799770 6349 ts_tablet_manager.cc:1905] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:30.800232 6203 tablet_service.cc:1467] Processing CreateTablet for tablet 687dad077c644725a044e6d22deca842 (DEFAULT_TABLE table=test-table [id=882ebbe85cee4ed687d5d77af84bbab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:30.800359 6203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687dad077c644725a044e6d22deca842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:30.800486 6076 tablet_service.cc:1467] Processing CreateTablet for tablet 687dad077c644725a044e6d22deca842 (DEFAULT_TABLE table=test-table [id=882ebbe85cee4ed687d5d77af84bbab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:30.800576 6076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687dad077c644725a044e6d22deca842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:30.801826 6349 ts_tablet_manager.cc:1918] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:30.801864 6308 tablet_bootstrap.cc:492] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579: Bootstrap starting.
I20260924 07:18:30.801892 6349 log.cc:1196] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/14e7f0d9cf104ed194145994d1c1d2de
I20260924 07:18:30.802196 6308 tablet_bootstrap.cc:654] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:30.802196 6349 ts_tablet_manager.cc:1939] T 14e7f0d9cf104ed194145994d1c1d2de P babe220450db4a5691db9ab7de4574cb: Deleting consensus metadata
I20260924 07:18:30.802457 5936 tablet_service.cc:1467] Processing CreateTablet for tablet 687dad077c644725a044e6d22deca842 (DEFAULT_TABLE table=test-table [id=882ebbe85cee4ed687d5d77af84bbab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:30.802513 6307 tablet_bootstrap.cc:492] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb: Bootstrap starting.
I20260924 07:18:30.802567 5936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687dad077c644725a044e6d22deca842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:30.802796 6350 tablet_replica.cc:330] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc: stopping tablet replica
I20260924 07:18:30.802901 6350 raft_consensus.cc:2229] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:30.802907 6307 tablet_bootstrap.cc:654] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:30.802954 6350 raft_consensus.cc:2258] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:30.803335 6350 ts_tablet_manager.cc:1905] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:30.804033 6305 tablet_bootstrap.cc:492] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc: Bootstrap starting.
I20260924 07:18:30.804082 5786 catalog_manager.cc:4841] TS babe220450db4a5691db9ab7de4574cb (127.3.246.66:35021): tablet 14e7f0d9cf104ed194145994d1c1d2de (table test-table [id=c24cf0ac1cdf42289b17ef50dc395c2a]) successfully deleted
I20260924 07:18:30.804440 6305 tablet_bootstrap.cc:654] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:30.804824 6307 tablet_bootstrap.cc:492] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb: No bootstrap required, opened a new log
I20260924 07:18:30.804880 6307 ts_tablet_manager.cc:1397] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:30.804930 6350 ts_tablet_manager.cc:1918] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:30.805007 6350 log.cc:1196] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/wal/wals/14e7f0d9cf104ed194145994d1c1d2de
I20260924 07:18:30.805033 6307 raft_consensus.cc:348] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.805110 6307 raft_consensus.cc:374] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:30.805137 6307 raft_consensus.cc:729] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babe220450db4a5691db9ab7de4574cb, State: Initialized, Role: FOLLOWER
I20260924 07:18:30.805182 6307 consensus_queue.cc:260] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.805285 6307 ts_tablet_manager.cc:1428] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:30.805588 6308 tablet_bootstrap.cc:492] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579: No bootstrap required, opened a new log
I20260924 07:18:30.805665 6308 ts_tablet_manager.cc:1397] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 07:18:30.805835 6308 raft_consensus.cc:348] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.805908 6308 raft_consensus.cc:374] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:30.805932 6308 raft_consensus.cc:729] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9158526bb6744c89df80d37e38a5579, State: Initialized, Role: FOLLOWER
I20260924 07:18:30.805974 6308 consensus_queue.cc:260] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.806067 6308 ts_tablet_manager.cc:1428] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:30.806180 6350 ts_tablet_manager.cc:1939] T 14e7f0d9cf104ed194145994d1c1d2de P 3b3909ef3d5d421aacb4364e76f4f2cc: Deleting consensus metadata
I20260924 07:18:30.806563 6305 tablet_bootstrap.cc:492] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc: No bootstrap required, opened a new log
I20260924 07:18:30.806636 6305 ts_tablet_manager.cc:1397] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 07:18:30.806705 5783 catalog_manager.cc:4841] TS 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821): tablet 14e7f0d9cf104ed194145994d1c1d2de (table test-table [id=c24cf0ac1cdf42289b17ef50dc395c2a]) successfully deleted
I20260924 07:18:30.806784 6305 raft_consensus.cc:348] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.806855 6305 raft_consensus.cc:374] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:30.806857 6351 tablet_replica.cc:330] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579: stopping tablet replica
I20260924 07:18:30.806881 6305 raft_consensus.cc:729] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3909ef3d5d421aacb4364e76f4f2cc, State: Initialized, Role: FOLLOWER
I20260924 07:18:30.806923 6305 consensus_queue.cc:260] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.806958 6351 raft_consensus.cc:2229] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:30.807013 6305 ts_tablet_manager.cc:1428] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:30.807018 6351 raft_consensus.cc:2258] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:30.807360 6351 ts_tablet_manager.cc:1905] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:30.809063 6351 ts_tablet_manager.cc:1918] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:30.809144 6351 log.cc:1196] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/14e7f0d9cf104ed194145994d1c1d2de
I20260924 07:18:30.809420 6351 ts_tablet_manager.cc:1939] T 14e7f0d9cf104ed194145994d1c1d2de P c9158526bb6744c89df80d37e38a5579: Deleting consensus metadata
I20260924 07:18:30.809971 5785 catalog_manager.cc:4841] TS c9158526bb6744c89df80d37e38a5579 (127.3.246.67:38227): tablet 14e7f0d9cf104ed194145994d1c1d2de (table test-table [id=c24cf0ac1cdf42289b17ef50dc395c2a]) successfully deleted
I20260924 07:18:30.819641 6347 raft_consensus.cc:482] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:30.819725 6347 raft_consensus.cc:504] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.819912 6347 leader_election.cc:288] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821), c9158526bb6744c89df80d37e38a5579 (127.3.246.67:38227)
I20260924 07:18:30.820070 6236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687dad077c644725a044e6d22deca842" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9158526bb6744c89df80d37e38a5579" is_pre_election: true
I20260924 07:18:30.820081 5956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687dad077c644725a044e6d22deca842" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" is_pre_election: true
I20260924 07:18:30.820166 6236 raft_consensus.cc:2454] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 0.
I20260924 07:18:30.820168 5956 raft_consensus.cc:2454] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 0.
I20260924 07:18:30.820328 6029 leader_election.cc:302] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: babe220450db4a5691db9ab7de4574cb, c9158526bb6744c89df80d37e38a5579; no voters:
I20260924 07:18:30.820422 6347 raft_consensus.cc:2781] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:30.820456 6347 raft_consensus.cc:482] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:30.820483 6347 raft_consensus.cc:3037] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:30.821342 6347 raft_consensus.cc:504] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.821483 6347 leader_election.cc:288] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 election: Requested vote from peers 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821), c9158526bb6744c89df80d37e38a5579 (127.3.246.67:38227)
I20260924 07:18:30.821609 5956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687dad077c644725a044e6d22deca842" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc"
I20260924 07:18:30.821707 5956 raft_consensus.cc:3037] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:30.821697 6236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687dad077c644725a044e6d22deca842" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9158526bb6744c89df80d37e38a5579"
I20260924 07:18:30.821754 6236 raft_consensus.cc:3037] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:30.822434 6236 raft_consensus.cc:2454] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 1.
I20260924 07:18:30.822433 5956 raft_consensus.cc:2454] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 1.
I20260924 07:18:30.822598 6027 leader_election.cc:302] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b3909ef3d5d421aacb4364e76f4f2cc, babe220450db4a5691db9ab7de4574cb; no voters:
I20260924 07:18:30.822708 6347 raft_consensus.cc:2781] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:30.822781 6347 raft_consensus.cc:686] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [term 1 LEADER]: Becoming Leader. State: Replica: babe220450db4a5691db9ab7de4574cb, State: Running, Role: LEADER
I20260924 07:18:30.822830 6347 consensus_queue.cc:237] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:30.823408 5798 catalog_manager.cc:5495] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb reported cstate change: term changed from 0 to 1, leader changed from <none> to babe220450db4a5691db9ab7de4574cb (127.3.246.66). New cstate: current_term: 1 leader_uuid: "babe220450db4a5691db9ab7de4574cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } health_report { overall_health: UNKNOWN } } }
I20260924 07:18:30.830366 5956 raft_consensus.cc:1261] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Refusing update from remote peer babe220450db4a5691db9ab7de4574cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:30.830417 6236 raft_consensus.cc:1261] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Refusing update from remote peer babe220450db4a5691db9ab7de4574cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:30.830615 6346 consensus_queue.cc:1035] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:30.830732 6347 consensus_queue.cc:1035] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:18:30.834975 6331 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 07:18:32.352923 5799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:32.353037 5799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:32.353823 5799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df: Sending DeleteTablet for 3 replicas of tablet 687dad077c644725a044e6d22deca842
I20260924 07:18:32.354087 5936 tablet_service.cc:1514] Processing DeleteTablet for tablet 687dad077c644725a044e6d22deca842 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:32 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:39764
I20260924 07:18:32.354100 6075 tablet_service.cc:1514] Processing DeleteTablet for tablet 687dad077c644725a044e6d22deca842 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:32 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:57544
I20260924 07:18:32.354105 6203 tablet_service.cc:1514] Processing DeleteTablet for tablet 687dad077c644725a044e6d22deca842 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:32 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:47624
I20260924 07:18:32.354307 6378 tablet_replica.cc:330] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb: stopping tablet replica
I20260924 07:18:32.354370 6378 raft_consensus.cc:2229] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:32.354421 6378 raft_consensus.cc:2258] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:32.354684 6378 ts_tablet_manager.cc:1905] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:32.356721 6378 ts_tablet_manager.cc:1918] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:18:32.356789 6378 log.cc:1196] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/687dad077c644725a044e6d22deca842
I20260924 07:18:32.357052 6378 ts_tablet_manager.cc:1939] T 687dad077c644725a044e6d22deca842 P babe220450db4a5691db9ab7de4574cb: Deleting consensus metadata
I20260924 07:18:32.357105 6380 tablet_replica.cc:330] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579: stopping tablet replica
I20260924 07:18:32.357177 6380 raft_consensus.cc:2229] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:32.357220 6380 raft_consensus.cc:2258] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:32.357220 6379 tablet_replica.cc:330] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc: stopping tablet replica
I20260924 07:18:32.357290 6379 raft_consensus.cc:2229] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:32.357337 6379 raft_consensus.cc:2258] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:32.357465 6380 ts_tablet_manager.cc:1905] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:32.357612 5786 catalog_manager.cc:4841] TS babe220450db4a5691db9ab7de4574cb (127.3.246.66:35021): tablet 687dad077c644725a044e6d22deca842 (table test-table [id=882ebbe85cee4ed687d5d77af84bbab8]) successfully deleted
I20260924 07:18:32.357630 6379 ts_tablet_manager.cc:1905] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:32.358850 6380 ts_tablet_manager.cc:1918] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:18:32.358927 6380 log.cc:1196] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/687dad077c644725a044e6d22deca842
I20260924 07:18:32.359136 6380 ts_tablet_manager.cc:1939] T 687dad077c644725a044e6d22deca842 P c9158526bb6744c89df80d37e38a5579: Deleting consensus metadata
I20260924 07:18:32.359596 6379 ts_tablet_manager.cc:1918] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:18:32.359659 6379 log.cc:1196] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/wal/wals/687dad077c644725a044e6d22deca842
I20260924 07:18:32.359643 5785 catalog_manager.cc:4841] TS c9158526bb6744c89df80d37e38a5579 (127.3.246.67:38227): tablet 687dad077c644725a044e6d22deca842 (table test-table [id=882ebbe85cee4ed687d5d77af84bbab8]) successfully deleted
I20260924 07:18:32.359871 6379 ts_tablet_manager.cc:1939] T 687dad077c644725a044e6d22deca842 P 3b3909ef3d5d421aacb4364e76f4f2cc: Deleting consensus metadata
I20260924 07:18:32.360210 5783 catalog_manager.cc:4841] TS 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821): tablet 687dad077c644725a044e6d22deca842 (table test-table [id=882ebbe85cee4ed687d5d77af84bbab8]) successfully deleted
Sep 24 07:18:32 dist-test-slave-rc1w krb5kdc[5754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234312, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:18:32 dist-test-slave-rc1w krb5kdc[5754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234312, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:32.378667 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:32.371147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.3.246.126:36791)
0924 07:18:32.371302 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:32.371304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:32.371371 (+ 67us) server_negotiation.cc:407] Connection header received
0924 07:18:32.371479 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:32.371480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:32.371493 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:32.371579 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:32.372150 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.372687 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.373472 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.373747 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.373839 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:32.376100 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:32.376101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:32.376101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:32.376107 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:32.377391 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:32.377856 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:32.377856 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:32.377857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:32.377927 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:32.378156 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:32.378157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:32.378157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:32.378263 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:32.378339 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:32.378527 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 07:18:32.378598 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260924 07:18:32.380165 5799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:32.380333 5799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:32.381654 5936 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcabad8be934bb7b06736a695ed125f (DEFAULT_TABLE table=test-table [id=e6c9cff3369d4dd699b0f0874d08e6a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:32.381659 6075 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcabad8be934bb7b06736a695ed125f (DEFAULT_TABLE table=test-table [id=e6c9cff3369d4dd699b0f0874d08e6a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:32.381805 6203 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcabad8be934bb7b06736a695ed125f (DEFAULT_TABLE table=test-table [id=e6c9cff3369d4dd699b0f0874d08e6a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:32.381845 5936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fcabad8be934bb7b06736a695ed125f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:32.381844 6075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fcabad8be934bb7b06736a695ed125f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:32.381939 6203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fcabad8be934bb7b06736a695ed125f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:32.383165 6389 tablet_bootstrap.cc:492] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579: Bootstrap starting.
I20260924 07:18:32.383308 6390 tablet_bootstrap.cc:492] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc: Bootstrap starting.
I20260924 07:18:32.383378 6391 tablet_bootstrap.cc:492] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb: Bootstrap starting.
I20260924 07:18:32.383692 6389 tablet_bootstrap.cc:654] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:32.383693 6390 tablet_bootstrap.cc:654] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:32.383734 6391 tablet_bootstrap.cc:654] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:32.384322 6389 tablet_bootstrap.cc:492] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579: No bootstrap required, opened a new log
I20260924 07:18:32.384321 6390 tablet_bootstrap.cc:492] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc: No bootstrap required, opened a new log
I20260924 07:18:32.384373 6390 ts_tablet_manager.cc:1397] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:32.384382 6389 ts_tablet_manager.cc:1397] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:32.384524 6389 raft_consensus.cc:348] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.384523 6390 raft_consensus.cc:348] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.384599 6389 raft_consensus.cc:374] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:32.384599 6390 raft_consensus.cc:374] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:32.384621 6390 raft_consensus.cc:729] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3909ef3d5d421aacb4364e76f4f2cc, State: Initialized, Role: FOLLOWER
I20260924 07:18:32.384621 6389 raft_consensus.cc:729] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9158526bb6744c89df80d37e38a5579, State: Initialized, Role: FOLLOWER
I20260924 07:18:32.384663 6390 consensus_queue.cc:260] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.384663 6389 consensus_queue.cc:260] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.384827 6389 ts_tablet_manager.cc:1428] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:32.384828 6390 ts_tablet_manager.cc:1428] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:32.385454 6391 tablet_bootstrap.cc:492] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb: No bootstrap required, opened a new log
I20260924 07:18:32.385509 6391 ts_tablet_manager.cc:1397] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:32.385627 6391 raft_consensus.cc:348] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.385730 6391 raft_consensus.cc:374] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:32.385766 6391 raft_consensus.cc:729] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babe220450db4a5691db9ab7de4574cb, State: Initialized, Role: FOLLOWER
I20260924 07:18:32.385831 6391 consensus_queue.cc:260] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.385947 6391 ts_tablet_manager.cc:1428] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:32.629940 6396 raft_consensus.cc:482] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:32.630043 6396 raft_consensus.cc:504] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.630465 6396 leader_election.cc:288] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821), babe220450db4a5691db9ab7de4574cb (127.3.246.66:35021)
I20260924 07:18:32.633564 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:32.630658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:40821 (local address 127.3.246.67:54123)
0924 07:18:32.630868 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:32.630881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:32.630944 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:32.631115 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:32.631116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:32.631119 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:32.631199 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:32.631201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.631913 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.631914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:32.632720 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:32.632722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.633282 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.633283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:32.633367 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:32.633370 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:32.633432 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 07:18:32.633474 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":128,"threads_started":1}
I20260924 07:18:32.633561 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:32.630748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:54123 (local address 127.3.246.65:40821)
0924 07:18:32.630915 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:32.630917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:32.630928 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:18:32.631027 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:32.631028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:32.631038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:32.631113 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:32.631274 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.631823 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.632812 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.633202 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.633285 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:32.633294 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:32.633450 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 07:18:32.633486 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260924 07:18:32.633853 5956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcabad8be934bb7b06736a695ed125f" candidate_uuid: "c9158526bb6744c89df80d37e38a5579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" is_pre_election: true
I20260924 07:18:32.633966 5956 raft_consensus.cc:2454] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9158526bb6744c89df80d37e38a5579 in term 0.
I20260924 07:18:32.634119 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:32.630802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:58547 (local address 127.3.246.66:35021)
0924 07:18:32.630943 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:32.630945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:32.631016 (+ 71us) server_negotiation.cc:407] Connection header received
0924 07:18:32.631087 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:32.631087 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:32.631096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:32.631169 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:32.631602 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.632077 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.633126 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.633637 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.633787 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:32.633798 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:32.633987 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 07:18:32.634043 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260924 07:18:32.634124 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:32.630725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:35021 (local address 127.3.246.67:58547)
0924 07:18:32.630932 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:32.630943 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:32.631011 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:32.631172 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:32.631173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:32.631175 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:32.631511 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:32.631513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.632159 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.632159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:32.633042 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:32.633044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.633829 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.633830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:32.633904 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:32.633907 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:32.633966 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 07:18:32.634006 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":141,"threads_started":1}
I20260924 07:18:32.634145 6167 leader_election.cc:302] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b3909ef3d5d421aacb4364e76f4f2cc, c9158526bb6744c89df80d37e38a5579; no voters:
I20260924 07:18:32.634284 6396 raft_consensus.cc:2781] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:32.634320 6396 raft_consensus.cc:482] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:32.634368 6396 raft_consensus.cc:3037] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:32.634503 6096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcabad8be934bb7b06736a695ed125f" candidate_uuid: "c9158526bb6744c89df80d37e38a5579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babe220450db4a5691db9ab7de4574cb" is_pre_election: true
I20260924 07:18:32.634651 6096 raft_consensus.cc:2454] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9158526bb6744c89df80d37e38a5579 in term 0.
I20260924 07:18:32.635165 6396 raft_consensus.cc:504] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.635315 6396 leader_election.cc:288] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [CANDIDATE]: Term 1 election: Requested vote from peers 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821), babe220450db4a5691db9ab7de4574cb (127.3.246.66:35021)
I20260924 07:18:32.635480 5956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcabad8be934bb7b06736a695ed125f" candidate_uuid: "c9158526bb6744c89df80d37e38a5579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc"
I20260924 07:18:32.635497 6096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcabad8be934bb7b06736a695ed125f" candidate_uuid: "c9158526bb6744c89df80d37e38a5579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babe220450db4a5691db9ab7de4574cb"
I20260924 07:18:32.635547 5956 raft_consensus.cc:3037] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:32.635569 6096 raft_consensus.cc:3037] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:32.636111 5956 raft_consensus.cc:2454] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9158526bb6744c89df80d37e38a5579 in term 1.
I20260924 07:18:32.636289 6096 raft_consensus.cc:2454] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9158526bb6744c89df80d37e38a5579 in term 1.
I20260924 07:18:32.636309 6167 leader_election.cc:302] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b3909ef3d5d421aacb4364e76f4f2cc, c9158526bb6744c89df80d37e38a5579; no voters:
I20260924 07:18:32.636402 6396 raft_consensus.cc:2781] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:32.636565 6396 raft_consensus.cc:686] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [term 1 LEADER]: Becoming Leader. State: Replica: c9158526bb6744c89df80d37e38a5579, State: Running, Role: LEADER
I20260924 07:18:32.636641 6396 consensus_queue.cc:237] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.637331 5798 catalog_manager.cc:5495] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9158526bb6744c89df80d37e38a5579 (127.3.246.67). New cstate: current_term: 1 leader_uuid: "c9158526bb6744c89df80d37e38a5579" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } health_report { overall_health: HEALTHY } } }
I20260924 07:18:32.670761 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:32.668180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47634 (local address 127.3.246.67:38227)
0924 07:18:32.668341 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:32.668343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:32.668350 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:18:32.668387 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:32.668387 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:32.668408 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:32.668479 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:32.668596 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.669079 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.669936 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.670156 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.670239 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:32.670342 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:32.670475 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:32.670540 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:32.670585 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 07:18:32.670603 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260924 07:18:32.675370 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:32.673260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:40101 (local address 127.3.246.66:35021)
0924 07:18:32.673333 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:32.673335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:32.673423 (+ 88us) server_negotiation.cc:407] Connection header received
0924 07:18:32.673471 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:32.673472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:32.673483 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:32.673527 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:32.673614 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.674072 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.674788 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.674974 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.675042 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:32.675184 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:32.675209 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:32.675249 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:32.675299 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 07:18:32.675318 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:18:32.675408 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:32.673188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:35021 (local address 127.3.246.67:40101)
0924 07:18:32.673371 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:32.673381 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:32.673432 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:32.673528 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:32.673528 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:32.673530 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:18:32.673573 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:32.673574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.674133 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.674134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:32.674692 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:32.674694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.675051 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.675052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:32.675100 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:32.675111 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:18:32.675259 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:18:32.675260 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 07:18:32.675291 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 07:18:32.675324 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260924 07:18:32.677486 5956 raft_consensus.cc:1261] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Refusing update from remote peer c9158526bb6744c89df80d37e38a5579: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:32.677626 6096 raft_consensus.cc:1261] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Refusing update from remote peer c9158526bb6744c89df80d37e38a5579: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:32.677786 6396 consensus_queue.cc:1035] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:32.677925 6401 consensus_queue.cc:1035] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [LEADER]: Connected to new peer: Peer: permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:32.685119 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:32.682780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:37771 (local address 127.3.246.67:38227)
0924 07:18:32.682816 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:32.682818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:32.682931 (+ 113us) server_negotiation.cc:407] Connection header received
0924 07:18:32.683001 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:32.683002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:32.683012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:32.683063 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:32.683164 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.683633 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.684389 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.684670 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.684756 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:32.684895 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:32.684927 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:32.684980 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:32.685029 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 07:18:32.685051 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:18:32.685148 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:32.682708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:38227 (local address 127.3.246.66:37771)
0924 07:18:32.682864 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:32.682875 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:32.682936 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:32.683066 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:32.683067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:32.683068 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:18:32.683115 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:32.683116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.683724 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.683726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:32.684298 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:32.684300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.684755 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.684756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:32.684815 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:32.684819 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:18:32.684981 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:18:32.684983 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:18:32.685028 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 07:18:32.685071 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":113,"threads_started":1}
I20260924 07:18:32.691677 5797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:32.691787 5797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:32.692406 5797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df: Sending DeleteTablet for 3 replicas of tablet 1fcabad8be934bb7b06736a695ed125f
I20260924 07:18:32.692663 6203 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fcabad8be934bb7b06736a695ed125f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:32 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:47624
I20260924 07:18:32.692664 5936 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fcabad8be934bb7b06736a695ed125f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:32 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:39764
I20260924 07:18:32.692759 6379 tablet_replica.cc:330] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc: stopping tablet replica
I20260924 07:18:32.692768 6380 tablet_replica.cc:330] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579: stopping tablet replica
I20260924 07:18:32.692801 6379 raft_consensus.cc:2229] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:32.692826 6380 raft_consensus.cc:2229] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:32.692839 6379 raft_consensus.cc:2258] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:32.692899 6380 raft_consensus.cc:2258] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:32.692916 6075 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fcabad8be934bb7b06736a695ed125f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:32 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:57544
I20260924 07:18:32.692998 6378 tablet_replica.cc:330] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb: stopping tablet replica
I20260924 07:18:32.693042 6378 raft_consensus.cc:2229] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:32.693070 6378 raft_consensus.cc:2258] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:32.693177 6380 ts_tablet_manager.cc:1905] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:32.693394 6378 ts_tablet_manager.cc:1905] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:32.693394 6379 ts_tablet_manager.cc:1905] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:32.694768 6380 ts_tablet_manager.cc:1918] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:32.694819 6380 log.cc:1196] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/1fcabad8be934bb7b06736a695ed125f
I20260924 07:18:32.694896 6379 ts_tablet_manager.cc:1918] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:32.694896 6378 ts_tablet_manager.cc:1918] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:32.694948 6379 log.cc:1196] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-0/wal/wals/1fcabad8be934bb7b06736a695ed125f
I20260924 07:18:32.694948 6378 log.cc:1196] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/1fcabad8be934bb7b06736a695ed125f
I20260924 07:18:32.695034 6380 ts_tablet_manager.cc:1939] T 1fcabad8be934bb7b06736a695ed125f P c9158526bb6744c89df80d37e38a5579: Deleting consensus metadata
I20260924 07:18:32.695168 6379 ts_tablet_manager.cc:1939] T 1fcabad8be934bb7b06736a695ed125f P 3b3909ef3d5d421aacb4364e76f4f2cc: Deleting consensus metadata
I20260924 07:18:32.695246 6378 ts_tablet_manager.cc:1939] T 1fcabad8be934bb7b06736a695ed125f P babe220450db4a5691db9ab7de4574cb: Deleting consensus metadata
I20260924 07:18:32.695559 5786 catalog_manager.cc:4841] TS babe220450db4a5691db9ab7de4574cb (127.3.246.66:35021): tablet 1fcabad8be934bb7b06736a695ed125f (table test-table [id=e6c9cff3369d4dd699b0f0874d08e6a0]) successfully deleted
I20260924 07:18:32.695571 5785 catalog_manager.cc:4841] TS c9158526bb6744c89df80d37e38a5579 (127.3.246.67:38227): tablet 1fcabad8be934bb7b06736a695ed125f (table test-table [id=e6c9cff3369d4dd699b0f0874d08e6a0]) successfully deleted
I20260924 07:18:32.695749 5783 catalog_manager.cc:4841] TS 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821): tablet 1fcabad8be934bb7b06736a695ed125f (table test-table [id=e6c9cff3369d4dd699b0f0874d08e6a0]) successfully deleted
I20260924 07:18:32.695873 5797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:32.696012 5797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:32.697094 5936 tablet_service.cc:1467] Processing CreateTablet for tablet 38523d74b47e4fe5b0f92081646c3317 (DEFAULT_TABLE table=test-table [id=c4fe56c1f9cc4600b446e7d54e9fc3df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:32.697126 6203 tablet_service.cc:1467] Processing CreateTablet for tablet 38523d74b47e4fe5b0f92081646c3317 (DEFAULT_TABLE table=test-table [id=c4fe56c1f9cc4600b446e7d54e9fc3df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:32.697170 6075 tablet_service.cc:1467] Processing CreateTablet for tablet 38523d74b47e4fe5b0f92081646c3317 (DEFAULT_TABLE table=test-table [id=c4fe56c1f9cc4600b446e7d54e9fc3df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:32.697204 5936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38523d74b47e4fe5b0f92081646c3317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:32.697225 6203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38523d74b47e4fe5b0f92081646c3317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:32.697254 6075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38523d74b47e4fe5b0f92081646c3317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:32.698458 6390 tablet_bootstrap.cc:492] T 38523d74b47e4fe5b0f92081646c3317 P 3b3909ef3d5d421aacb4364e76f4f2cc: Bootstrap starting.
I20260924 07:18:32.698464 6389 tablet_bootstrap.cc:492] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579: Bootstrap starting.
I20260924 07:18:32.698813 6389 tablet_bootstrap.cc:654] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:32.699127 6390 tablet_bootstrap.cc:654] T 38523d74b47e4fe5b0f92081646c3317 P 3b3909ef3d5d421aacb4364e76f4f2cc: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:32.699489 6391 tablet_bootstrap.cc:492] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb: Bootstrap starting.
I20260924 07:18:32.699489 6389 tablet_bootstrap.cc:492] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579: No bootstrap required, opened a new log
I20260924 07:18:32.699536 6389 ts_tablet_manager.cc:1397] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:32.699699 6389 raft_consensus.cc:348] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.699779 6389 raft_consensus.cc:374] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:32.699803 6389 raft_consensus.cc:729] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9158526bb6744c89df80d37e38a5579, State: Initialized, Role: FOLLOWER
I20260924 07:18:32.699820 6391 tablet_bootstrap.cc:654] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:32.699853 6389 consensus_queue.cc:260] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.699985 6389 ts_tablet_manager.cc:1428] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:32.700358 6391 tablet_bootstrap.cc:492] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb: No bootstrap required, opened a new log
I20260924 07:18:32.700425 6391 ts_tablet_manager.cc:1397] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:32.700533 6391 raft_consensus.cc:348] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.700627 6391 raft_consensus.cc:374] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:32.700652 6391 raft_consensus.cc:729] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babe220450db4a5691db9ab7de4574cb, State: Initialized, Role: FOLLOWER
I20260924 07:18:32.700695 6391 consensus_queue.cc:260] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.700790 6391 ts_tablet_manager.cc:1428] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:32.701047 6390 tablet_bootstrap.cc:492] T 38523d74b47e4fe5b0f92081646c3317 P 3b3909ef3d5d421aacb4364e76f4f2cc: No bootstrap required, opened a new log
I20260924 07:18:32.701094 6390 ts_tablet_manager.cc:1397] T 38523d74b47e4fe5b0f92081646c3317 P 3b3909ef3d5d421aacb4364e76f4f2cc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 07:18:32.701256 6390 raft_consensus.cc:348] T 38523d74b47e4fe5b0f92081646c3317 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.701333 6390 raft_consensus.cc:374] T 38523d74b47e4fe5b0f92081646c3317 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:32.701354 6390 raft_consensus.cc:729] T 38523d74b47e4fe5b0f92081646c3317 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3909ef3d5d421aacb4364e76f4f2cc, State: Initialized, Role: FOLLOWER
I20260924 07:18:32.701390 6390 consensus_queue.cc:260] T 38523d74b47e4fe5b0f92081646c3317 P 3b3909ef3d5d421aacb4364e76f4f2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.701479 6390 ts_tablet_manager.cc:1428] T 38523d74b47e4fe5b0f92081646c3317 P 3b3909ef3d5d421aacb4364e76f4f2cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:32.802323 6366 raft_consensus.cc:482] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:32.802459 6366 raft_consensus.cc:504] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.802651 6366 leader_election.cc:288] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821), c9158526bb6744c89df80d37e38a5579 (127.3.246.67:38227)
I20260924 07:18:32.802953 6236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38523d74b47e4fe5b0f92081646c3317" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9158526bb6744c89df80d37e38a5579" is_pre_election: true
I20260924 07:18:32.802930 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38523d74b47e4fe5b0f92081646c3317" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" is_pre_election: true
I20260924 07:18:32.803047 5955 raft_consensus.cc:2454] T 38523d74b47e4fe5b0f92081646c3317 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 0.
I20260924 07:18:32.803047 6236 raft_consensus.cc:2454] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 0.
I20260924 07:18:32.803208 6029 leader_election.cc:302] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: babe220450db4a5691db9ab7de4574cb, c9158526bb6744c89df80d37e38a5579; no voters:
I20260924 07:18:32.803354 6366 raft_consensus.cc:2781] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:32.803418 6366 raft_consensus.cc:482] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:32.803445 6366 raft_consensus.cc:3037] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:32.804431 6366 raft_consensus.cc:504] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.804601 6366 leader_election.cc:288] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 election: Requested vote from peers 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821), c9158526bb6744c89df80d37e38a5579 (127.3.246.67:38227)
I20260924 07:18:32.804762 6236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38523d74b47e4fe5b0f92081646c3317" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9158526bb6744c89df80d37e38a5579"
I20260924 07:18:32.804781 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38523d74b47e4fe5b0f92081646c3317" candidate_uuid: "babe220450db4a5691db9ab7de4574cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc"
I20260924 07:18:32.804862 5955 raft_consensus.cc:3037] T 38523d74b47e4fe5b0f92081646c3317 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:32.804865 6236 raft_consensus.cc:3037] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:32.805502 6236 raft_consensus.cc:2454] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 1.
I20260924 07:18:32.805707 6029 leader_election.cc:302] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: babe220450db4a5691db9ab7de4574cb, c9158526bb6744c89df80d37e38a5579; no voters:
I20260924 07:18:32.805769 5955 raft_consensus.cc:2454] T 38523d74b47e4fe5b0f92081646c3317 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate babe220450db4a5691db9ab7de4574cb in term 1.
I20260924 07:18:32.805842 6366 raft_consensus.cc:2781] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:32.805891 6366 raft_consensus.cc:686] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [term 1 LEADER]: Becoming Leader. State: Replica: babe220450db4a5691db9ab7de4574cb, State: Running, Role: LEADER
I20260924 07:18:32.805943 6366 consensus_queue.cc:237] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } }
I20260924 07:18:32.806634 5798 catalog_manager.cc:5495] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb reported cstate change: term changed from 0 to 1, leader changed from <none> to babe220450db4a5691db9ab7de4574cb (127.3.246.66). New cstate: current_term: 1 leader_uuid: "babe220450db4a5691db9ab7de4574cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "babe220450db4a5691db9ab7de4574cb" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 } health_report { overall_health: UNKNOWN } } }
I20260924 07:18:32.819667 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:32.817756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.3.246.66:35021)
0924 07:18:32.817796 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:32.817798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:32.817807 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:32.817833 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:32.817834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:32.817847 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:32.817890 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:32.817974 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.818432 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.819081 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:32.819252 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:32.819300 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:32.819408 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:32.819431 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:32.819464 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:32.819598 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 07:18:32.819613 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:18:32.821890 5955 raft_consensus.cc:1261] T 38523d74b47e4fe5b0f92081646c3317 P 3b3909ef3d5d421aacb4364e76f4f2cc [term 1 FOLLOWER]: Refusing update from remote peer babe220450db4a5691db9ab7de4574cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:32.821904 6236 raft_consensus.cc:1261] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Refusing update from remote peer babe220450db4a5691db9ab7de4574cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:32.822108 6313 consensus_queue.cc:1035] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b3909ef3d5d421aacb4364e76f4f2cc" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:32.822235 6361 consensus_queue.cc:1035] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9158526bb6744c89df80d37e38a5579" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:18:32.826426 6359 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 07:18:34.343225 5798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:34.343369 5798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:34.344210 5798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d4ac125a02c454986f03c59d11853df: Sending DeleteTablet for 3 replicas of tablet 38523d74b47e4fe5b0f92081646c3317
I20260924 07:18:34.344491 5936 tablet_service.cc:1514] Processing DeleteTablet for tablet 38523d74b47e4fe5b0f92081646c3317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:34 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:39764
I20260924 07:18:34.344491 6203 tablet_service.cc:1514] Processing DeleteTablet for tablet 38523d74b47e4fe5b0f92081646c3317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:34 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:47624
I20260924 07:18:34.344503 6075 tablet_service.cc:1514] Processing DeleteTablet for tablet 38523d74b47e4fe5b0f92081646c3317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:34 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:57544
I20260924 07:18:34.344712 6442 tablet_replica.cc:330] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb: stopping tablet replica
I20260924 07:18:34.344786 6442 raft_consensus.cc:2229] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:34.344816 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 5871
I20260924 07:18:34.344851 6442 raft_consensus.cc:2258] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:34.346730 6442 ts_tablet_manager.cc:1905] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:34.348752 6442 ts_tablet_manager.cc:1918] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:18:34.348819 6442 log.cc:1196] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/38523d74b47e4fe5b0f92081646c3317
I20260924 07:18:34.349068 6442 ts_tablet_manager.cc:1939] T 38523d74b47e4fe5b0f92081646c3317 P babe220450db4a5691db9ab7de4574cb: Deleting consensus metadata
I20260924 07:18:34.349613 5786 catalog_manager.cc:4841] TS babe220450db4a5691db9ab7de4574cb (127.3.246.66:35021): tablet 38523d74b47e4fe5b0f92081646c3317 (table test-table [id=c4fe56c1f9cc4600b446e7d54e9fc3df]) successfully deleted
I20260924 07:18:34.351017 6440 tablet_replica.cc:330] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579: stopping tablet replica
I20260924 07:18:34.351081 6440 raft_consensus.cc:2229] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:34.351133 6440 raft_consensus.cc:2258] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 07:18:34.351344 5783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:40821 (error 108)
I20260924 07:18:34.351383 6440 ts_tablet_manager.cc:1905] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:34.351961 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 6011
I20260924 07:18:34.352018 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:34.351735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:40821 (local address 127.0.0.1:39766)
0924 07:18:34.351933 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:34.351967 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
W20260924 07:18:34.352102 5783 catalog_manager.cc:4584] TS 3b3909ef3d5d421aacb4364e76f4f2cc (127.3.246.65:40821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38523d74b47e4fe5b0f92081646c3317: Network error: Client connection negotiation failed: client connection to 127.3.246.65:40821: connect: Connection refused (error 111)
I20260924 07:18:34.353989 6440 ts_tablet_manager.cc:1918] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:18:34.354054 6440 log.cc:1196] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/38523d74b47e4fe5b0f92081646c3317
I20260924 07:18:34.354279 6440 ts_tablet_manager.cc:1939] T 38523d74b47e4fe5b0f92081646c3317 P c9158526bb6744c89df80d37e38a5579: Deleting consensus metadata
I20260924 07:18:34.354745 5785 catalog_manager.cc:4841] TS c9158526bb6744c89df80d37e38a5579 (127.3.246.67:38227): tablet 38523d74b47e4fe5b0f92081646c3317 (table test-table [id=c4fe56c1f9cc4600b446e7d54e9fc3df]) successfully deleted
I20260924 07:18:34.358659 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 6151
I20260924 07:18:34.363937 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 5767
2026-09-24T07:18:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5952 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:34 dist-test-slave-rc1w krb5kdc[6447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:34 dist-test-slave-rc1w krb5kdc[6447](info): set up 2 sockets
Sep 24 07:18:34 dist-test-slave-rc1w krb5kdc[6447](info): commencing operation
krb5kdc: starting...
W20260924 07:18:35.413489 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:35 dist-test-slave-rc1w krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:18:35.532958 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:33201
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:33201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:35.613690 6460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:35.613955 6460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:35.613993 6460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:35.615154 6460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:35.615226 6460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:35.615253 6460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:35.615285 6460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:35.615309 6460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:35.616794 6460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:33201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:33201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:35.617090 6460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:35.617326 6460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:35.622246 6466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:35.622256 6468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:35.622306 6465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:35.622617 6460 server_base.cc:1034] running on GCE node
I20260924 07:18:35.622849 6460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:35.623124 6460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:35.624275 6460 hybrid_clock.cc:648] HybridClock initialized: now 1790234315624261 us; error 32 us; skew 500 ppm
Sep 24 07:18:35 dist-test-slave-rc1w krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:35.636298 6460 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:18:35.636869 6460 webserver.cc:460] Webserver started at http://127.3.246.126:45479/ using document root <none> and password file <none>
I20260924 07:18:35.637138 6460 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:35.637213 6460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:35.637368 6460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:35.638670 6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "e15d5bff0e48460ca94f090e6e448841"
format_stamp: "Formatted at 2026-09-24 07:18:35 on dist-test-slave-rc1w"
server_key: "1be3ba073c9e4cf1165c3811308fcdaa"
server_key_iv: "cbdf55fbed679410d3b6ea83b131fc16"
server_key_version: "encryptionkey@0"
I20260924 07:18:35.639137 6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "e15d5bff0e48460ca94f090e6e448841"
format_stamp: "Formatted at 2026-09-24 07:18:35 on dist-test-slave-rc1w"
server_key: "1be3ba073c9e4cf1165c3811308fcdaa"
server_key_iv: "cbdf55fbed679410d3b6ea83b131fc16"
server_key_version: "encryptionkey@0"
I20260924 07:18:35.640547 6460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:35.641358 6475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:35.641561 6460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:18:35.641705 6460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "e15d5bff0e48460ca94f090e6e448841"
format_stamp: "Formatted at 2026-09-24 07:18:35 on dist-test-slave-rc1w"
server_key: "1be3ba073c9e4cf1165c3811308fcdaa"
server_key_iv: "cbdf55fbed679410d3b6ea83b131fc16"
server_key_version: "encryptionkey@0"
I20260924 07:18:35.641799 6460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:35.661828 6460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:35.663666 6460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:35.663827 6460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:35.668004 6460 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:33201
I20260924 07:18:35.668038 6527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:33201 every 8 connection(s)
I20260924 07:18:35.668449 6460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:35.669309 6528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:35.671609 6528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841: Bootstrap starting.
I20260924 07:18:35.672197 6528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:35.672508 6528 log.cc:824] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:35.673197 6528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841: No bootstrap required, opened a new log
I20260924 07:18:35.674898 6528 raft_consensus.cc:348] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15d5bff0e48460ca94f090e6e448841" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33201 } }
I20260924 07:18:35.675032 6528 raft_consensus.cc:374] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:35.675055 6528 raft_consensus.cc:729] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15d5bff0e48460ca94f090e6e448841, State: Initialized, Role: FOLLOWER
I20260924 07:18:35.675204 6528 consensus_queue.cc:260] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15d5bff0e48460ca94f090e6e448841" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33201 } }
I20260924 07:18:35.675279 6528 raft_consensus.cc:388] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:35.675321 6528 raft_consensus.cc:482] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:35.675381 6528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:35.675927 6528 raft_consensus.cc:504] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15d5bff0e48460ca94f090e6e448841" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33201 } }
I20260924 07:18:35.676010 6528 leader_election.cc:302] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e15d5bff0e48460ca94f090e6e448841; no voters:
I20260924 07:18:35.676204 6528 leader_election.cc:288] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:35.676249 6531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:35.676473 6528 sys_catalog.cc:564] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:35.676658 6531 raft_consensus.cc:686] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [term 1 LEADER]: Becoming Leader. State: Replica: e15d5bff0e48460ca94f090e6e448841, State: Running, Role: LEADER
I20260924 07:18:35.676808 6531 consensus_queue.cc:237] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15d5bff0e48460ca94f090e6e448841" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33201 } }
I20260924 07:18:35.677209 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e15d5bff0e48460ca94f090e6e448841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15d5bff0e48460ca94f090e6e448841" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33201 } } }
I20260924 07:18:35.677232 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e15d5bff0e48460ca94f090e6e448841. Latest consensus state: current_term: 1 leader_uuid: "e15d5bff0e48460ca94f090e6e448841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15d5bff0e48460ca94f090e6e448841" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33201 } } }
I20260924 07:18:35.677321 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:35.677340 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:35.677539 6540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:35.677944 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 6460
I20260924 07:18:35.678035 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:35.678225 4057 external_mini_cluster.cc:1428] Setting key 31c9902d16b466db3c76123b1aa5e780
I20260924 07:18:35.678735 6540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:35.680475 6540 catalog_manager.cc:1347] Generated new cluster ID: ebedb30edfdc4baa85536b462e64dbbb
I20260924 07:18:35.680558 6540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:18:35 dist-test-slave-rc1w krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:35.687458 6540 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:35.688201 6540 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:35.690071 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:35.678980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.3.246.126:33201)
0924 07:18:35.679139 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:35.679149 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:35.679245 (+ 96us) server_negotiation.cc:407] Connection header received
0924 07:18:35.679401 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:35.679409 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:35.679463 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:35.679629 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:35.680359 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:35.681778 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:35.682860 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:35.683494 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:35.683586 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:35.686555 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:35.686559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:35.686560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:35.686571 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:35.688489 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:35.688853 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:35.688857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:35.688857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:35.688957 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:35.689166 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:35.689167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:35.689167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:35.689404 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:35.689565 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:35.689692 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 07:18:35.689850 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260924 07:18:35.698714 6540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e15d5bff0e48460ca94f090e6e448841: Generated new TSK 0
I20260924 07:18:35.698943 6540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:18:35.734447 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:33201
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:35.817183 6554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:35.817422 6554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:35.817447 6554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:35.818796 6554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:35.818840 6554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:35.818868 6554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:35.820304 6554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:33201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:35.820495 6554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:35.820672 6554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:35.825968 6559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:35.825977 6562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:35.825977 6560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:35.826325 6554 server_base.cc:1034] running on GCE node
I20260924 07:18:35.826541 6554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:35.826761 6554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:35.827943 6554 hybrid_clock.cc:648] HybridClock initialized: now 1790234315827931 us; error 31 us; skew 500 ppm
Sep 24 07:18:35 dist-test-slave-rc1w krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:35.830572 6554 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:18:35.831071 6554 webserver.cc:460] Webserver started at http://127.3.246.65:38385/ using document root <none> and password file <none>
I20260924 07:18:35.831321 6554 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:35.831390 6554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:35.831544 6554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:35.832605 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "5332551ffb3f4ec39d6bae10455acc44"
format_stamp: "Formatted at 2026-09-24 07:18:35 on dist-test-slave-rc1w"
server_key: "479e59fcaec8307030b64304e20ca3b3"
server_key_iv: "1f400a23a9802557959586b68f30ed7e"
server_key_version: "encryptionkey@0"
I20260924 07:18:35.832980 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "5332551ffb3f4ec39d6bae10455acc44"
format_stamp: "Formatted at 2026-09-24 07:18:35 on dist-test-slave-rc1w"
server_key: "479e59fcaec8307030b64304e20ca3b3"
server_key_iv: "1f400a23a9802557959586b68f30ed7e"
server_key_version: "encryptionkey@0"
I20260924 07:18:35.834404 6554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:35.835207 6569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:35.835433 6554 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:35.835538 6554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "5332551ffb3f4ec39d6bae10455acc44"
format_stamp: "Formatted at 2026-09-24 07:18:35 on dist-test-slave-rc1w"
server_key: "479e59fcaec8307030b64304e20ca3b3"
server_key_iv: "1f400a23a9802557959586b68f30ed7e"
server_key_version: "encryptionkey@0"
I20260924 07:18:35.835611 6554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:35.847069 6554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:35.848910 6554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:35.849095 6554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:35.849323 6554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:35.849629 6554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:35.849664 6554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:35.849776 6554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:35.849809 6554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:35.855657 6554 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:43363
I20260924 07:18:35.855700 6682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:43363 every 8 connection(s)
I20260924 07:18:35.856016 6554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:35.859359 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 6554
I20260924 07:18:35.859436 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:35.859625 4057 external_mini_cluster.cc:1428] Setting key 6db473d684e21a5a1a9c692ec8268999
Sep 24 07:18:35 dist-test-slave-rc1w krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:35.866107 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:35.856752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:43617 (local address 127.3.246.126:33201)
0924 07:18:35.856834 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:35.856836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:35.857014 (+ 178us) server_negotiation.cc:407] Connection header received
0924 07:18:35.857484 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:35.857485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:35.857501 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:35.857556 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:35.858579 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:35.859083 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:35.860307 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:35.860516 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:35.860602 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:35.863112 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:35.863113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:35.863114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:35.863122 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:35.864590 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:35.865066 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:35.865068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:35.865068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:35.865130 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:35.865398 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:35.865399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:35.865399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:35.865517 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:35.865593 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:35.865927 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 07:18:35.866015 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:18:35.866310 6685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:35.856712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:33201 (local address 127.3.246.65:43617)
0924 07:18:35.856891 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:35.856928 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:35.857401 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:35.857621 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:35.857625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:35.858259 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:35.858488 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:35.858494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:35.859164 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:35.859164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:35.860214 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:35.860217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:35.860614 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:35.860615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:35.860754 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:35.861279 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:35.861287 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:35.863011 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:35.864733 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:35.864740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:35.864743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:35.864969 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:35.865213 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:35.865214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:35.865215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:35.865328 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:35.865586 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:35.865593 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:35.865791 (+ 198us) client_negotiation.cc:769] Sending connection context
0924 07:18:35.865905 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 07:18:35.866060 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260924 07:18:35.866773 6683 heartbeater.cc:347] Connected to a master server at 127.3.246.126:33201
I20260924 07:18:35.866863 6683 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:35.867046 6683 heartbeater.cc:511] Master 127.3.246.126:33201 requested a full tablet report, sending...
I20260924 07:18:35.867604 6492 ts_manager.cc:194] Registered new tserver with Master: 5332551ffb3f4ec39d6bae10455acc44 (127.3.246.65:43363)
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
I20260924 07:18:35.868386 6492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:43617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:18:35.885273 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:33201
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:35.966320 6688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:35.966522 6688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:35.966544 6688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:35.967646 6688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:35.967684 6688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:35.967710 6688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:35.969110 6688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:33201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:35.969287 6688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:35.969533 6688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:35.974236 6696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:35.974257 6693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:35.974283 6694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:35.974529 6688 server_base.cc:1034] running on GCE node
I20260924 07:18:35.974721 6688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:35.974927 6688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:35.976086 6688 hybrid_clock.cc:648] HybridClock initialized: now 1790234315976073 us; error 27 us; skew 500 ppm
Sep 24 07:18:35 dist-test-slave-rc1w krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:35.978695 6688 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:18:35.979127 6688 webserver.cc:460] Webserver started at http://127.3.246.66:33313/ using document root <none> and password file <none>
I20260924 07:18:35.979311 6688 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:35.979350 6688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:35.979434 6688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:35.980590 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "7962e1e1bf424d87bcef8cc6daf594a7"
format_stamp: "Formatted at 2026-09-24 07:18:35 on dist-test-slave-rc1w"
server_key: "da02116ed5a51093894475e7b42aace4"
server_key_iv: "1d14269318f95cad5a644b446645261f"
server_key_version: "encryptionkey@0"
I20260924 07:18:35.980902 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "7962e1e1bf424d87bcef8cc6daf594a7"
format_stamp: "Formatted at 2026-09-24 07:18:35 on dist-test-slave-rc1w"
server_key: "da02116ed5a51093894475e7b42aace4"
server_key_iv: "1d14269318f95cad5a644b446645261f"
server_key_version: "encryptionkey@0"
I20260924 07:18:35.982234 6688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:35.982879 6703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:35.983053 6688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:18:35.983160 6688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "7962e1e1bf424d87bcef8cc6daf594a7"
format_stamp: "Formatted at 2026-09-24 07:18:35 on dist-test-slave-rc1w"
server_key: "da02116ed5a51093894475e7b42aace4"
server_key_iv: "1d14269318f95cad5a644b446645261f"
server_key_version: "encryptionkey@0"
I20260924 07:18:35.983233 6688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:36.016309 6688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:36.018769 6688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:36.018954 6688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:36.019181 6688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:36.019495 6688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:36.019553 6688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:36.019601 6688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:36.019629 6688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:36.025628 6688 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:44995
I20260924 07:18:36.025727 6816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:44995 every 8 connection(s)
I20260924 07:18:36.026050 6688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:36.030246 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 6688
I20260924 07:18:36.030364 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:36.030543 4057 external_mini_cluster.cc:1428] Setting key f0283b44ff8f3ab9a36e5fcd9e0086ce
Sep 24 07:18:36 dist-test-slave-rc1w krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:36.035537 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:36.026821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:39151 (local address 127.3.246.126:33201)
0924 07:18:36.026892 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:36.026894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:36.027068 (+ 174us) server_negotiation.cc:407] Connection header received
0924 07:18:36.027548 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:36.027549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:36.027558 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:36.027626 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:36.028497 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:36.028985 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:36.030031 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:36.030220 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:36.030291 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:36.032613 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:36.032614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:36.032615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:36.032622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:36.034054 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:36.034480 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:36.034482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:36.034482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:36.034542 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:36.034892 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:36.034892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:36.034893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:36.034993 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:36.035046 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:36.035405 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 07:18:36.035474 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:18:36.035871 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:36.026792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:33201 (local address 127.3.246.66:39151)
0924 07:18:36.026947 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:36.026987 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:36.027468 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:36.027682 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:36.027685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:36.028230 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:36.028420 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:36.028425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:36.029064 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:36.029064 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:36.029929 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:36.029931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:36.030313 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:36.030315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:36.030470 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:36.031003 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:36.031013 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:36.032500 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:36.034153 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:36.034160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:36.034162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:36.034386 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:36.034644 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:36.034646 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:36.034646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:36.034815 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:36.035061 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:36.035068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:36.035252 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 07:18:36.035375 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 07:18:36.035576 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260924 07:18:36.036438 6817 heartbeater.cc:347] Connected to a master server at 127.3.246.126:33201
I20260924 07:18:36.036545 6817 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:36.036758 6817 heartbeater.cc:511] Master 127.3.246.126:33201 requested a full tablet report, sending...
I20260924 07:18:36.037145 6492 ts_manager.cc:194] Registered new tserver with Master: 7962e1e1bf424d87bcef8cc6daf594a7 (127.3.246.66:44995)
I20260924 07:18:36.037554 6492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:39151
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:18:36.057181 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:33201
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:36.139154 6822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:36.139355 6822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:36.139379 6822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:36.140599 6822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:36.140643 6822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:36.140695 6822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:36.142162 6822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:33201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:36.142407 6822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:36.142639 6822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:36.147854 6830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:36.147909 6828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:36.148067 6822 server_base.cc:1034] running on GCE node
W20260924 07:18:36.147886 6827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:36.148399 6822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:36.148615 6822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:36.149775 6822 hybrid_clock.cc:648] HybridClock initialized: now 1790234316149753 us; error 42 us; skew 500 ppm
Sep 24 07:18:36 dist-test-slave-rc1w krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234316, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:36.152277 6822 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:18:36.152654 6822 webserver.cc:460] Webserver started at http://127.3.246.67:41239/ using document root <none> and password file <none>
I20260924 07:18:36.152845 6822 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:36.152889 6822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:36.152979 6822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:36.154137 6822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "35834ace89a747c282595ec2d9bbb0dc"
format_stamp: "Formatted at 2026-09-24 07:18:36 on dist-test-slave-rc1w"
server_key: "ecab76962919cb48a6f9f2ce22d9fe3b"
server_key_iv: "2e8bca414ea96c67ae29da4a068d0676"
server_key_version: "encryptionkey@0"
I20260924 07:18:36.154445 6822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "35834ace89a747c282595ec2d9bbb0dc"
format_stamp: "Formatted at 2026-09-24 07:18:36 on dist-test-slave-rc1w"
server_key: "ecab76962919cb48a6f9f2ce22d9fe3b"
server_key_iv: "2e8bca414ea96c67ae29da4a068d0676"
server_key_version: "encryptionkey@0"
I20260924 07:18:36.155719 6822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:36.156473 6837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:36.156652 6822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:36.156739 6822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "35834ace89a747c282595ec2d9bbb0dc"
format_stamp: "Formatted at 2026-09-24 07:18:36 on dist-test-slave-rc1w"
server_key: "ecab76962919cb48a6f9f2ce22d9fe3b"
server_key_iv: "2e8bca414ea96c67ae29da4a068d0676"
server_key_version: "encryptionkey@0"
I20260924 07:18:36.156796 6822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:36.168442 6822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:36.171094 6822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:36.171268 6822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:36.171517 6822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:36.171886 6822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:36.171937 6822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:36.171968 6822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:36.171988 6822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:36.178299 6822 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:44373
I20260924 07:18:36.178354 6950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:44373 every 8 connection(s)
I20260924 07:18:36.178680 6822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:36.182262 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 6822
I20260924 07:18:36.182343 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:36.182550 4057 external_mini_cluster.cc:1428] Setting key c6815cbc0333e1628cd3d8e408f3d411
Sep 24 07:18:36 dist-test-slave-rc1w krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234316, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:36.188503 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:36.179491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:50505 (local address 127.3.246.126:33201)
0924 07:18:36.179555 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:36.179571 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:36.179750 (+ 179us) server_negotiation.cc:407] Connection header received
0924 07:18:36.180298 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:36.180299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:36.180312 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:36.180380 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:36.181320 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:36.181921 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:36.183055 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:36.183255 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:36.183339 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:36.185394 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:36.185395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:36.185395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:36.185403 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:36.186786 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:36.187343 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:36.187344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:36.187345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:36.187437 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:36.187750 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:36.187751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:36.187752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:36.187919 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:36.187993 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:36.188320 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 07:18:36.188412 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:18:36.188699 6953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:36.179464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:33201 (local address 127.3.246.67:50505)
0924 07:18:36.179626 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:36.179668 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:36.180185 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:36.180450 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:36.180454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:36.181032 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:36.181220 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:36.181224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:36.182009 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:36.182011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:36.182957 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:36.182960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:36.183349 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:36.183350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:36.183473 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:36.183912 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:36.183919 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:36.185293 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:36.186884 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:36.186890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:36.186892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:36.187216 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:36.187506 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:36.187507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:36.187507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:36.187676 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:36.188001 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:36.188008 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:36.188169 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 07:18:36.188279 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 07:18:36.188461 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260924 07:18:36.189213 6951 heartbeater.cc:347] Connected to a master server at 127.3.246.126:33201
I20260924 07:18:36.189293 6951 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:36.189443 6951 heartbeater.cc:511] Master 127.3.246.126:33201 requested a full tablet report, sending...
I20260924 07:18:36.189790 6492 ts_manager.cc:194] Registered new tserver with Master: 35834ace89a747c282595ec2d9bbb0dc (127.3.246.67:44373)
I20260924 07:18:36.190120 6492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:50505
I20260924 07:18:36.194309 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:18:36.205904 6550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:36.204897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.3.246.126:33201)
0924 07:18:36.204972 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:36.204976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:36.205081 (+ 105us) server_negotiation.cc:407] Connection header received
0924 07:18:36.205149 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:36.205149 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:36.205159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:36.205218 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:36.205826 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57728: BlockingRecv error: recv got EOF from 127.0.0.1:57728 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:18:36.206764 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 6554
I20260924 07:18:36.211314 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 6688
I20260924 07:18:36.215346 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 6822
I20260924 07:18:36.219551 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 6460
2026-09-24T07:18:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1854 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:36 dist-test-slave-rc1w krb5kdc[6963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:36 dist-test-slave-rc1w krb5kdc[6963](info): set up 2 sockets
Sep 24 07:18:36 dist-test-slave-rc1w krb5kdc[6963](info): commencing operation
krb5kdc: starting...
W20260924 07:18:37.257584 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:37 dist-test-slave-rc1w krb5kdc[6963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:18:37.376518 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:45827
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:45827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:37.457089 6976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:37.457288 6976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:37.457311 6976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:37.458452 6976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:37.458489 6976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:37.458503 6976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:37.458515 6976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:37.458529 6976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:37.459926 6976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:45827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:45827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:37.460143 6976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:37.460379 6976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:18:37.465273 6976 server_base.cc:1034] running on GCE node
W20260924 07:18:37.465240 6984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:37.465240 6982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:37.465225 6981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:37.465770 6976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:37.466075 6976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:37.467221 6976 hybrid_clock.cc:648] HybridClock initialized: now 1790234317467200 us; error 35 us; skew 500 ppm
Sep 24 07:18:37 dist-test-slave-rc1w krb5kdc[6963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234317, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:37.469779 6976 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:18:37.470211 6976 webserver.cc:460] Webserver started at http://127.3.246.126:34825/ using document root <none> and password file <none>
I20260924 07:18:37.470436 6976 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:37.470482 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:37.470633 6976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:37.471727 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "f6214d7c809d4eac815b6c7934191975"
format_stamp: "Formatted at 2026-09-24 07:18:37 on dist-test-slave-rc1w"
server_key: "f59621169f719d983791fc6c737dfa2f"
server_key_iv: "497651be9c2a06a7d8531073082bd8bc"
server_key_version: "encryptionkey@0"
I20260924 07:18:37.472097 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "f6214d7c809d4eac815b6c7934191975"
format_stamp: "Formatted at 2026-09-24 07:18:37 on dist-test-slave-rc1w"
server_key: "f59621169f719d983791fc6c737dfa2f"
server_key_iv: "497651be9c2a06a7d8531073082bd8bc"
server_key_version: "encryptionkey@0"
I20260924 07:18:37.473899 6976 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260924 07:18:37.474613 6991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:37.474813 6976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:37.474908 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "f6214d7c809d4eac815b6c7934191975"
format_stamp: "Formatted at 2026-09-24 07:18:37 on dist-test-slave-rc1w"
server_key: "f59621169f719d983791fc6c737dfa2f"
server_key_iv: "497651be9c2a06a7d8531073082bd8bc"
server_key_version: "encryptionkey@0"
I20260924 07:18:37.474963 6976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:37.489411 6976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:37.491391 6976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:37.491504 6976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:37.495426 6976 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:45827
I20260924 07:18:37.495499 7043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:45827 every 8 connection(s)
I20260924 07:18:37.495823 6976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:37.496837 7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:37.499215 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975: Bootstrap starting.
I20260924 07:18:37.499814 7044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:37.500113 7044 log.cc:824] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:37.500787 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975: No bootstrap required, opened a new log
I20260924 07:18:37.501087 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 6976
I20260924 07:18:37.501160 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:37.501353 4057 external_mini_cluster.cc:1428] Setting key dfbc0b3cb55bb7b21dbbd6465957d005
I20260924 07:18:37.503512 7044 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6214d7c809d4eac815b6c7934191975" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 45827 } }
I20260924 07:18:37.503688 7044 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:37.503726 7044 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6214d7c809d4eac815b6c7934191975, State: Initialized, Role: FOLLOWER
I20260924 07:18:37.503887 7044 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6214d7c809d4eac815b6c7934191975" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 45827 } }
I20260924 07:18:37.503970 7044 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:37.503994 7044 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:37.504060 7044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:37.504667 7044 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6214d7c809d4eac815b6c7934191975" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 45827 } }
I20260924 07:18:37.504772 7044 leader_election.cc:302] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6214d7c809d4eac815b6c7934191975; no voters:
I20260924 07:18:37.504917 7044 leader_election.cc:288] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:37.504988 7049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:37.505169 7044 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:37.505177 7049 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [term 1 LEADER]: Becoming Leader. State: Replica: f6214d7c809d4eac815b6c7934191975, State: Running, Role: LEADER
I20260924 07:18:37.505339 7049 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6214d7c809d4eac815b6c7934191975" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 45827 } }
I20260924 07:18:37.505748 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6214d7c809d4eac815b6c7934191975. Latest consensus state: current_term: 1 leader_uuid: "f6214d7c809d4eac815b6c7934191975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6214d7c809d4eac815b6c7934191975" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 45827 } } }
I20260924 07:18:37.505897 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:37.506140 7053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:37.505728 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6214d7c809d4eac815b6c7934191975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6214d7c809d4eac815b6c7934191975" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 45827 } } }
I20260924 07:18:37.506533 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:37.506690 7053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:37.509577 7053 catalog_manager.cc:1347] Generated new cluster ID: 7ae1b94cae6b4538ba5d2e4fa53d4a74
I20260924 07:18:37.509641 7053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:18:37 dist-test-slave-rc1w krb5kdc[6963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:37.514789 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:37.501913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.3.246.126:45827)
0924 07:18:37.502112 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:37.502122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:37.502160 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:18:37.503476 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:37.503486 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:37.503555 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:37.503723 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:37.504527 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:37.505936 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:37.507205 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:37.507802 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:37.507909 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:37.510649 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:37.510661 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:37.510661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:37.510673 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:37.512671 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:37.513047 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:37.513050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:37.513051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:37.513537 (+ 486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:37.513757 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:37.513758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:37.513759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:37.514079 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:37.514243 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:37.514323 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 07:18:37.514562 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260924 07:18:37.516027 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:18:37.535106 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:37.530803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.3.246.126:45827)
0924 07:18:37.530844 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:37.530846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:37.530995 (+ 149us) server_negotiation.cc:407] Connection header received
0924 07:18:37.531069 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:37.531070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:37.531081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:37.531151 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:37.531571 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:37.532294 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:37.533361 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:37.533692 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:37.533749 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:37.534549 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:37.534551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:37.534551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:37.534819 (+ 268us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:37.534841 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:37.534849 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:37.534913 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:37.534989 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 07:18:37.535014 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:18:37.544654 7053 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:37.545313 7053 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:37.553855 7053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6214d7c809d4eac815b6c7934191975: Generated new TSK 0
I20260924 07:18:37.554068 7053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:18:37.561563 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 6976
2026-09-24T07:18:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1340 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:37 dist-test-slave-rc1w krb5kdc[7077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:37 dist-test-slave-rc1w krb5kdc[7077](info): set up 2 sockets
Sep 24 07:18:37 dist-test-slave-rc1w krb5kdc[7077](info): commencing operation
krb5kdc: starting...
W20260924 07:18:38.589735 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:38 dist-test-slave-rc1w krb5kdc[7077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:18:38.708473 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37671
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:37671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:38.788849 7090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:38.789083 7090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:38.789119 7090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:38.790261 7090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:38.790333 7090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:38.790359 7090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:38.790387 7090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:38.790416 7090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:38.791853 7090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:37671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:38.792120 7090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:38.792338 7090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:38.797032 7098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:38.797120 7090 server_base.cc:1034] running on GCE node
W20260924 07:18:38.798744 7095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:38.798856 7096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:38.799162 7090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:38.799463 7090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:38.800640 7090 hybrid_clock.cc:648] HybridClock initialized: now 1790234318800627 us; error 34 us; skew 500 ppm
Sep 24 07:18:38 dist-test-slave-rc1w krb5kdc[7077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234318, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:38.803201 7090 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:18:38.803633 7090 webserver.cc:460] Webserver started at http://127.3.246.126:39195/ using document root <none> and password file <none>
I20260924 07:18:38.803860 7090 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:38.803929 7090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:38.804080 7090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:38.805150 7090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "c3e04e80e7e24ca8aff7d8cb56571741"
format_stamp: "Formatted at 2026-09-24 07:18:38 on dist-test-slave-rc1w"
server_key: "7fd8e9b1b14ba3c44c0fdd54d5a0e051"
server_key_iv: "35e1cbd562768c2f3e477db2efd7434c"
server_key_version: "encryptionkey@0"
I20260924 07:18:38.805508 7090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "c3e04e80e7e24ca8aff7d8cb56571741"
format_stamp: "Formatted at 2026-09-24 07:18:38 on dist-test-slave-rc1w"
server_key: "7fd8e9b1b14ba3c44c0fdd54d5a0e051"
server_key_iv: "35e1cbd562768c2f3e477db2efd7434c"
server_key_version: "encryptionkey@0"
I20260924 07:18:38.806928 7090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:38.807639 7105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:38.807803 7090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:18:38.807914 7090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "c3e04e80e7e24ca8aff7d8cb56571741"
format_stamp: "Formatted at 2026-09-24 07:18:38 on dist-test-slave-rc1w"
server_key: "7fd8e9b1b14ba3c44c0fdd54d5a0e051"
server_key_iv: "35e1cbd562768c2f3e477db2efd7434c"
server_key_version: "encryptionkey@0"
I20260924 07:18:38.808001 7090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:38.827512 7090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:38.829758 7090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:38.829936 7090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:38.833835 7090 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:37671
I20260924 07:18:38.833870 7157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:37671 every 8 connection(s)
I20260924 07:18:38.834219 7090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:38.835373 7158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:38.837637 7158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741: Bootstrap starting.
I20260924 07:18:38.838243 7158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:38.838521 7158 log.cc:824] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:38.839293 7158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741: No bootstrap required, opened a new log
I20260924 07:18:38.840895 7158 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e04e80e7e24ca8aff7d8cb56571741" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37671 } }
I20260924 07:18:38.841033 7158 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:38.841058 7158 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e04e80e7e24ca8aff7d8cb56571741, State: Initialized, Role: FOLLOWER
I20260924 07:18:38.841231 7158 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e04e80e7e24ca8aff7d8cb56571741" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37671 } }
I20260924 07:18:38.841296 7158 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:38.841318 7158 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:38.841346 7158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:38.842227 7158 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e04e80e7e24ca8aff7d8cb56571741" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37671 } }
I20260924 07:18:38.842316 7158 leader_election.cc:302] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3e04e80e7e24ca8aff7d8cb56571741; no voters:
I20260924 07:18:38.842506 7158 leader_election.cc:288] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:38.842607 7161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:38.842751 7158 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:38.842819 7161 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [term 1 LEADER]: Becoming Leader. State: Replica: c3e04e80e7e24ca8aff7d8cb56571741, State: Running, Role: LEADER
I20260924 07:18:38.842948 7161 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e04e80e7e24ca8aff7d8cb56571741" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37671 } }
I20260924 07:18:38.843202 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 7090
I20260924 07:18:38.843318 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:38.843425 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3e04e80e7e24ca8aff7d8cb56571741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e04e80e7e24ca8aff7d8cb56571741" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37671 } } }
I20260924 07:18:38.843504 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:38.843533 4057 external_mini_cluster.cc:1428] Setting key 55f2c39b9b6189ee6625f77eff8aca7b
I20260924 07:18:38.843689 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3e04e80e7e24ca8aff7d8cb56571741. Latest consensus state: current_term: 1 leader_uuid: "c3e04e80e7e24ca8aff7d8cb56571741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e04e80e7e24ca8aff7d8cb56571741" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37671 } } }
I20260924 07:18:38.843968 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741 [sys.catalog]: This master's current role is: LEADER
W20260924 07:18:38.844645 7178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:18:38.844717 7178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:18:38.844808 7170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:38.845024 7170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:38.846599 7170 catalog_manager.cc:1347] Generated new cluster ID: c57adfcee145434cb127b5f267fc6da4
I20260924 07:18:38.846653 7170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:18:38 dist-test-slave-rc1w krb5kdc[7077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:38.853147 7170 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:38.853848 7170 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:38.854740 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:38.844309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.3.246.126:37671)
0924 07:18:38.844474 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:38.844487 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:38.844522 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:18:38.844646 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:38.844653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:38.844708 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:38.844846 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:38.845735 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:38.846979 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:38.847805 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:38.848265 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:38.848404 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:38.851030 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:38.851034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:38.851034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:38.851049 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:38.852950 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:38.853412 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:38.853416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:38.853416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:38.853514 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:38.853829 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:38.853830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:38.853831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:38.854110 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:38.854322 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:38.854395 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 07:18:38.854567 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260924 07:18:38.862775 7170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3e04e80e7e24ca8aff7d8cb56571741: Generated new TSK 0
I20260924 07:18:38.862944 7170 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:18:38.888307 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37671
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:38.967373 7184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:38.967566 7184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:38.967589 7184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:38.968721 7184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:38.968758 7184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:38.968784 7184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:38.970266 7184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:38.970501 7184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:38.970726 7184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:38.975601 7192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:38.975608 7189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:38.975608 7190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:38.975991 7184 server_base.cc:1034] running on GCE node
I20260924 07:18:38.976264 7184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:38.976501 7184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:38.977664 7184 hybrid_clock.cc:648] HybridClock initialized: now 1790234318977652 us; error 33 us; skew 500 ppm
Sep 24 07:18:38 dist-test-slave-rc1w krb5kdc[7077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234318, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:38.980119 7184 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:18:38.980643 7184 webserver.cc:460] Webserver started at http://127.3.246.65:44509/ using document root <none> and password file <none>
I20260924 07:18:38.980873 7184 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:38.980940 7184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:38.981074 7184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:38.982173 7184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "fe4dff2bc4f941edbc7f2e2ac6a22b61"
format_stamp: "Formatted at 2026-09-24 07:18:38 on dist-test-slave-rc1w"
server_key: "32f0b6131400267713b4d847c88697bd"
server_key_iv: "e930ea84e671d51d7af6624ca03101c3"
server_key_version: "encryptionkey@0"
I20260924 07:18:38.982537 7184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "fe4dff2bc4f941edbc7f2e2ac6a22b61"
format_stamp: "Formatted at 2026-09-24 07:18:38 on dist-test-slave-rc1w"
server_key: "32f0b6131400267713b4d847c88697bd"
server_key_iv: "e930ea84e671d51d7af6624ca03101c3"
server_key_version: "encryptionkey@0"
I20260924 07:18:38.983834 7184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:38.984508 7199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:38.984678 7184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:38.984774 7184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "fe4dff2bc4f941edbc7f2e2ac6a22b61"
format_stamp: "Formatted at 2026-09-24 07:18:38 on dist-test-slave-rc1w"
server_key: "32f0b6131400267713b4d847c88697bd"
server_key_iv: "e930ea84e671d51d7af6624ca03101c3"
server_key_version: "encryptionkey@0"
I20260924 07:18:38.984859 7184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:39.009428 7184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:39.011574 7184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:39.011739 7184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:39.011984 7184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:39.012292 7184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:39.012352 7184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:39.012396 7184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:39.012427 7184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:39.018287 7184 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:33963
I20260924 07:18:39.018338 7312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:33963 every 8 connection(s)
I20260924 07:18:39.018662 7184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:39.022936 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 7184
I20260924 07:18:39.023052 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:39.023247 4057 external_mini_cluster.cc:1428] Setting key 18da9c393e2a0c5d399ef26de2acbd97
Sep 24 07:18:39 dist-test-slave-rc1w krb5kdc[7077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234318, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:39.028609 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:39.019361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:35129 (local address 127.3.246.126:37671)
0924 07:18:39.019423 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:39.019425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:39.019636 (+ 211us) server_negotiation.cc:407] Connection header received
0924 07:18:39.020166 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:39.020167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:39.020180 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:39.020240 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:39.021176 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.021750 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.022752 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.022980 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.023063 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:39.025710 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:39.025711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:39.025712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:39.025720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:39.027197 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:39.027643 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:39.027644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:39.027645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:39.027708 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:39.027962 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:39.027963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:39.027964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:39.028077 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:39.028154 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:39.028439 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 07:18:39.028518 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:18:39.028789 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:39.019332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37671 (local address 127.3.246.65:35129)
0924 07:18:39.019516 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:39.019551 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:39.020061 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:39.020315 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:39.020319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:39.020885 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:39.021072 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:39.021076 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.021861 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.021862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:39.022659 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:39.022661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.023128 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.023129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:39.023245 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:39.023871 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:39.023880 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:39.025595 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:39.027314 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:39.027322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:39.027324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:39.027550 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:39.027777 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:39.027778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:39.027778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:39.027893 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:39.028145 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:39.028152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:39.028299 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 07:18:39.028403 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 07:18:39.028559 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260924 07:18:39.029304 7313 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37671
I20260924 07:18:39.029392 7313 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:39.029584 7313 heartbeater.cc:511] Master 127.3.246.126:37671 requested a full tablet report, sending...
I20260924 07:18:39.030181 7122 ts_manager.cc:194] Registered new tserver with Master: fe4dff2bc4f941edbc7f2e2ac6a22b61 (127.3.246.65:33963)
I20260924 07:18:39.031132 7122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:35129
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:18:39.048794 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37671
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:39.128104 7318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:39.128304 7318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:39.128327 7318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:39.129386 7318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:39.129419 7318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:39.129446 7318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:39.130812 7318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:39.131002 7318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:39.131165 7318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:39.136126 7324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:39.136137 7326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:39.136170 7318 server_base.cc:1034] running on GCE node
W20260924 07:18:39.136137 7323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:39.136476 7318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:39.136713 7318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:39.137877 7318 hybrid_clock.cc:648] HybridClock initialized: now 1790234319137857 us; error 35 us; skew 500 ppm
Sep 24 07:18:39 dist-test-slave-rc1w krb5kdc[7077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234319, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:39.140393 7318 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:18:39.140774 7318 webserver.cc:460] Webserver started at http://127.3.246.66:34657/ using document root <none> and password file <none>
I20260924 07:18:39.140980 7318 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:39.141022 7318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:39.141112 7318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:39.142191 7318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "77a3642613814581b73c3e63098c2680"
format_stamp: "Formatted at 2026-09-24 07:18:39 on dist-test-slave-rc1w"
server_key: "de397c63d07572badbd644ce1c132c7b"
server_key_iv: "7587f67a9db99b8439da2402295bd56f"
server_key_version: "encryptionkey@0"
I20260924 07:18:39.142496 7318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "77a3642613814581b73c3e63098c2680"
format_stamp: "Formatted at 2026-09-24 07:18:39 on dist-test-slave-rc1w"
server_key: "de397c63d07572badbd644ce1c132c7b"
server_key_iv: "7587f67a9db99b8439da2402295bd56f"
server_key_version: "encryptionkey@0"
I20260924 07:18:39.144032 7318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:39.144714 7333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:39.144918 7318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:39.144990 7318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "77a3642613814581b73c3e63098c2680"
format_stamp: "Formatted at 2026-09-24 07:18:39 on dist-test-slave-rc1w"
server_key: "de397c63d07572badbd644ce1c132c7b"
server_key_iv: "7587f67a9db99b8439da2402295bd56f"
server_key_version: "encryptionkey@0"
I20260924 07:18:39.145044 7318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:39.177518 7318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:39.179551 7318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:39.179666 7318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:39.179841 7318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:39.180176 7318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:39.180212 7318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:39.180235 7318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:39.180294 7318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:39.186498 7318 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:33659
I20260924 07:18:39.186543 7446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:33659 every 8 connection(s)
I20260924 07:18:39.186903 7318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:18:39 dist-test-slave-rc1w krb5kdc[7077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234319, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:39.193617 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 7318
I20260924 07:18:39.194031 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:39.194270 4057 external_mini_cluster.cc:1428] Setting key f4135649fa5f5890f1fc6ee436390651
I20260924 07:18:39.196624 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:39.187685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:59755 (local address 127.3.246.126:37671)
0924 07:18:39.187769 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:39.187771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:39.187957 (+ 186us) server_negotiation.cc:407] Connection header received
0924 07:18:39.188445 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:39.188447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:39.188456 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:39.188500 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:39.189498 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.190053 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.191067 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.191245 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.191295 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:39.193495 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:39.193497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:39.193497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:39.193505 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:39.195173 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:39.195596 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:39.195598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:39.195598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:39.195671 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:39.195919 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:39.195920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:39.195920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:39.196072 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:39.196161 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:39.196400 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 07:18:39.196492 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:18:39.196838 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:39.187637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37671 (local address 127.3.246.66:59755)
0924 07:18:39.187826 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:39.187871 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:39.188351 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:39.188572 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:39.188576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:39.189184 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:39.189407 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:39.189412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.190155 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.190157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:39.190989 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:39.190992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.191337 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.191338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:39.191452 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:39.192043 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:39.192052 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:39.193397 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:39.195260 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:39.195273 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:39.195281 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:39.195516 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:39.195757 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:39.195758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:39.195758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:39.195853 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:39.196152 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:39.196156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:39.196269 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 07:18:39.196350 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:18:39.196535 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":107,"threads_started":1}
I20260924 07:18:39.197304 7447 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37671
I20260924 07:18:39.197417 7447 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:39.197598 7447 heartbeater.cc:511] Master 127.3.246.126:37671 requested a full tablet report, sending...
I20260924 07:18:39.198060 7122 ts_manager.cc:194] Registered new tserver with Master: 77a3642613814581b73c3e63098c2680 (127.3.246.66:33659)
I20260924 07:18:39.198459 7122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:59755
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:18:39.220709 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37671
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:39.300941 7452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:39.301134 7452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:39.301157 7452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:39.302330 7452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:39.302373 7452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:39.302400 7452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:39.303747 7452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:39.303928 7452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:39.304092 7452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:39.309267 7458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:39.309302 7452 server_base.cc:1034] running on GCE node
W20260924 07:18:39.309314 7460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:39.309268 7457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:39.309793 7452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:39.310063 7452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:39.311213 7452 hybrid_clock.cc:648] HybridClock initialized: now 1790234319311181 us; error 49 us; skew 500 ppm
Sep 24 07:18:39 dist-test-slave-rc1w krb5kdc[7077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234319, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:39.313642 7452 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:18:39.314191 7452 webserver.cc:460] Webserver started at http://127.3.246.67:42889/ using document root <none> and password file <none>
I20260924 07:18:39.314383 7452 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:39.314424 7452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:39.314510 7452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:39.315521 7452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "35689572753f4f1eab56d252ad167b57"
format_stamp: "Formatted at 2026-09-24 07:18:39 on dist-test-slave-rc1w"
server_key: "b559ae0f6c789ef27417d9c020dce5f4"
server_key_iv: "ea9976fdc8a4b1d3afee9f59616a7426"
server_key_version: "encryptionkey@0"
I20260924 07:18:39.315822 7452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "35689572753f4f1eab56d252ad167b57"
format_stamp: "Formatted at 2026-09-24 07:18:39 on dist-test-slave-rc1w"
server_key: "b559ae0f6c789ef27417d9c020dce5f4"
server_key_iv: "ea9976fdc8a4b1d3afee9f59616a7426"
server_key_version: "encryptionkey@0"
I20260924 07:18:39.317106 7452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:18:39.318060 7467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:39.318259 7452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:39.318344 7452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "35689572753f4f1eab56d252ad167b57"
format_stamp: "Formatted at 2026-09-24 07:18:39 on dist-test-slave-rc1w"
server_key: "b559ae0f6c789ef27417d9c020dce5f4"
server_key_iv: "ea9976fdc8a4b1d3afee9f59616a7426"
server_key_version: "encryptionkey@0"
I20260924 07:18:39.318400 7452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:39.335072 7452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:39.337360 7452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:39.337482 7452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:39.337785 7452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:39.338068 7452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:39.338101 7452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:39.338124 7452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:39.338135 7452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:39.344069 7452 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:43331
I20260924 07:18:39.344136 7580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:43331 every 8 connection(s)
I20260924 07:18:39.344488 7452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:39.345468 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 7452
I20260924 07:18:39.345577 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:39.345847 4057 external_mini_cluster.cc:1428] Setting key 9f7384254652b4d85e3df3ea0af6cfde
Sep 24 07:18:39 dist-test-slave-rc1w krb5kdc[7077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234319, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:39.353588 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:39.345237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:60451 (local address 127.3.246.126:37671)
0924 07:18:39.345290 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:39.345292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:39.345459 (+ 167us) server_negotiation.cc:407] Connection header received
0924 07:18:39.346108 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:39.346109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:39.346122 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:39.346186 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:39.347130 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.347631 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.348754 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.348968 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.349041 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:39.351089 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:39.351090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:39.351091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:39.351097 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:39.352264 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:39.352655 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:39.352656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:39.352657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:39.352722 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:39.352961 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:39.352962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:39.352962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:39.353075 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:39.353153 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:39.353423 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 07:18:39.353501 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:18:39.353823 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:39.345194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37671 (local address 127.3.246.67:60451)
0924 07:18:39.345347 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:39.345386 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:39.345996 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:39.346258 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:39.346263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:39.346842 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:39.347024 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:39.347029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.347724 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.347726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:39.348649 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:39.348651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.349046 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.349047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:39.349183 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:39.349621 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:39.349629 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:39.350987 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:39.352363 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:39.352369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:39.352370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:39.352581 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:39.352788 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:39.352789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:39.352790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:39.352891 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:39.353142 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:39.353148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:39.353283 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 07:18:39.353392 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 07:18:39.353543 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260924 07:18:39.354219 7581 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37671
I20260924 07:18:39.354331 7581 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:39.354501 7581 heartbeater.cc:511] Master 127.3.246.126:37671 requested a full tablet report, sending...
I20260924 07:18:39.354913 7122 ts_manager.cc:194] Registered new tserver with Master: 35689572753f4f1eab56d252ad167b57 (127.3.246.67:43331)
I20260924 07:18:39.355285 7122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:60451
I20260924 07:18:39.357481 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:18:39 dist-test-slave-rc1w krb5kdc[7077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234319, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 07:18:39 dist-test-slave-rc1w krb5kdc[7077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234319, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:39.380393 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:39.372923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.3.246.126:37671)
0924 07:18:39.373006 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:39.373009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:39.373108 (+ 99us) server_negotiation.cc:407] Connection header received
0924 07:18:39.373211 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:39.373212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:39.373223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:39.373291 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:39.373949 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.374447 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.375212 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.375422 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.375500 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:39.377970 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:39.377972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:39.377972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:39.377981 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:39.379220 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:39.379562 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:39.379563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:39.379564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:39.379621 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:39.379826 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:39.379827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:39.379827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:39.379976 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:39.380091 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:39.380198 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 07:18:39.380281 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 07:18:39.380646 7122 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37426
I20260924 07:18:39.381163 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 7184
I20260924 07:18:39.386041 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 7318
I20260924 07:18:39.390436 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 7452
I20260924 07:18:39.394771 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 7090
2026-09-24T07:18:39Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1834 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:39 dist-test-slave-rc1w krb5kdc[7593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:39 dist-test-slave-rc1w krb5kdc[7593](info): set up 2 sockets
Sep 24 07:18:39 dist-test-slave-rc1w krb5kdc[7593](info): commencing operation
krb5kdc: starting...
W20260924 07:18:40.461800 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:40 dist-test-slave-rc1w krb5kdc[7593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:18:40.585400 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:41731
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:41731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:40.664707 7606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:40.664904 7606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:40.664927 7606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:40.666041 7606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:40.666074 7606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:40.666088 7606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:40.666101 7606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:40.666113 7606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:40.667555 7606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:41731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:41731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:40.667752 7606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:40.667920 7606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:40.672648 7614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:40.672667 7611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:40.672667 7612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:40.673046 7606 server_base.cc:1034] running on GCE node
I20260924 07:18:40.673336 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:40.673645 7606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:40.674832 7606 hybrid_clock.cc:648] HybridClock initialized: now 1790234320674814 us; error 35 us; skew 500 ppm
Sep 24 07:18:40 dist-test-slave-rc1w krb5kdc[7593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234320, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:40.688071 7606 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:18:40.688602 7606 webserver.cc:460] Webserver started at http://127.3.246.126:45527/ using document root <none> and password file <none>
I20260924 07:18:40.688850 7606 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:40.688920 7606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:40.689066 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:40.690258 7606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "95630968ca724366a9a67ab123b93059"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-rc1w"
server_key: "96ce37eabbde53d3a335d7b861ee3314"
server_key_iv: "d47d4a36c47cf0d374508cb40c18ec54"
server_key_version: "encryptionkey@0"
I20260924 07:18:40.690676 7606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "95630968ca724366a9a67ab123b93059"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-rc1w"
server_key: "96ce37eabbde53d3a335d7b861ee3314"
server_key_iv: "d47d4a36c47cf0d374508cb40c18ec54"
server_key_version: "encryptionkey@0"
I20260924 07:18:40.692009 7606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:40.692732 7621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:40.692917 7606 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:40.693019 7606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "95630968ca724366a9a67ab123b93059"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-rc1w"
server_key: "96ce37eabbde53d3a335d7b861ee3314"
server_key_iv: "d47d4a36c47cf0d374508cb40c18ec54"
server_key_version: "encryptionkey@0"
I20260924 07:18:40.693099 7606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:40.719745 7606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:40.721808 7606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:40.721980 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:40.726064 7606 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:41731
I20260924 07:18:40.726100 7673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:41731 every 8 connection(s)
I20260924 07:18:40.726624 7606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:40.727275 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:40.729502 7674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059: Bootstrap starting.
I20260924 07:18:40.730118 7674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:40.730392 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 7606
I20260924 07:18:40.730427 7674 log.cc:824] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:40.730499 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:40.730741 4057 external_mini_cluster.cc:1428] Setting key bce41dc091f479f9891ffd924bc4193e
I20260924 07:18:40.731261 7674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059: No bootstrap required, opened a new log
I20260924 07:18:40.733887 7674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95630968ca724366a9a67ab123b93059" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41731 } }
I20260924 07:18:40.734052 7674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:40.734083 7674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95630968ca724366a9a67ab123b93059, State: Initialized, Role: FOLLOWER
I20260924 07:18:40.734194 7674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95630968ca724366a9a67ab123b93059" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41731 } }
I20260924 07:18:40.734263 7674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:40.734289 7674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:40.734324 7674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:40.734989 7674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95630968ca724366a9a67ab123b93059" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41731 } }
I20260924 07:18:40.735077 7674 leader_election.cc:302] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95630968ca724366a9a67ab123b93059; no voters:
I20260924 07:18:40.735246 7674 leader_election.cc:288] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:40.735277 7679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:40.735407 7679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [term 1 LEADER]: Becoming Leader. State: Replica: 95630968ca724366a9a67ab123b93059, State: Running, Role: LEADER
I20260924 07:18:40.735487 7679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95630968ca724366a9a67ab123b93059" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41731 } }
I20260924 07:18:40.735515 7674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:40.735948 7681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95630968ca724366a9a67ab123b93059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95630968ca724366a9a67ab123b93059" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41731 } } }
I20260924 07:18:40.736063 7681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:40.735935 7679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95630968ca724366a9a67ab123b93059. Latest consensus state: current_term: 1 leader_uuid: "95630968ca724366a9a67ab123b93059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95630968ca724366a9a67ab123b93059" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41731 } } }
I20260924 07:18:40.736267 7679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:40.736685 7685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:40.737353 7685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 07:18:40 dist-test-slave-rc1w krb5kdc[7593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:40.739109 7685 catalog_manager.cc:1347] Generated new cluster ID: b0b8dcbb596241eaad40a3b32cc2b83d
I20260924 07:18:40.739161 7685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:40.742923 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:40.731252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.3.246.126:41731)
0924 07:18:40.731447 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:40.731457 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:40.731490 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:18:40.732459 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:40.732469 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:40.732532 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:40.732684 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:40.733297 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.734705 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.735767 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.736448 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.736565 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:40.739424 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:40.739428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:40.739428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:40.739439 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:40.741147 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:40.741519 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:40.741522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:40.741522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:40.741628 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:40.741891 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:40.741892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:40.741892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:40.742131 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:40.742300 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:40.742408 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 07:18:40.742544 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260924 07:18:40.751648 7685 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:40.752425 7685 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:40.755961 7685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95630968ca724366a9a67ab123b93059: Generated new TSK 0
I20260924 07:18:40.756121 7685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:18:40.784721 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:41731
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:40.865903 7700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:40.866140 7700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:40.866180 7700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:40.867398 7700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:40.867463 7700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:40.867507 7700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:40.869010 7700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:41731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:40.869272 7700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:40.869503 7700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:40.874409 7705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:40.874698 7700 server_base.cc:1034] running on GCE node
W20260924 07:18:40.874409 7708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:40.874411 7706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:40.875072 7700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:40.875279 7700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:40.876451 7700 hybrid_clock.cc:648] HybridClock initialized: now 1790234320876451 us; error 42 us; skew 500 ppm
Sep 24 07:18:40 dist-test-slave-rc1w krb5kdc[7593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234320, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:40.879029 7700 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:18:40.879477 7700 webserver.cc:460] Webserver started at http://127.3.246.65:37143/ using document root <none> and password file <none>
I20260924 07:18:40.879654 7700 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:40.879694 7700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:40.879781 7700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:40.880867 7700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "72512beea3ff48ee88a02728179e7577"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-rc1w"
server_key: "cefd77ed4c65bd1a81b2f5b064b42d3a"
server_key_iv: "3914fff05da5ea16aea3872b2e0043b5"
server_key_version: "encryptionkey@0"
I20260924 07:18:40.881171 7700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "72512beea3ff48ee88a02728179e7577"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-rc1w"
server_key: "cefd77ed4c65bd1a81b2f5b064b42d3a"
server_key_iv: "3914fff05da5ea16aea3872b2e0043b5"
server_key_version: "encryptionkey@0"
I20260924 07:18:40.882506 7700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:40.883149 7715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:40.883358 7700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:18:40.883461 7700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "72512beea3ff48ee88a02728179e7577"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-rc1w"
server_key: "cefd77ed4c65bd1a81b2f5b064b42d3a"
server_key_iv: "3914fff05da5ea16aea3872b2e0043b5"
server_key_version: "encryptionkey@0"
I20260924 07:18:40.883535 7700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:40.895543 7700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:40.898053 7700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:40.898221 7700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:40.898522 7700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:40.898880 7700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:40.898913 7700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:40.898993 7700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:40.899026 7700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:40.904707 7700 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:46737
I20260924 07:18:40.904842 7828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:46737 every 8 connection(s)
I20260924 07:18:40.905112 7700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:40.909497 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 7700
I20260924 07:18:40.909619 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:40.909833 4057 external_mini_cluster.cc:1428] Setting key e4d75dc7664f9730ab98df9a4e9e0710
Sep 24 07:18:40 dist-test-slave-rc1w krb5kdc[7593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234320, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:40.915606 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:40.905908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:43961 (local address 127.3.246.126:41731)
0924 07:18:40.905982 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:40.905984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:40.906176 (+ 192us) server_negotiation.cc:407] Connection header received
0924 07:18:40.907066 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:40.907067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:40.907077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:40.907141 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:40.907708 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.908210 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.909194 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.909477 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.909554 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:40.912669 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:40.912671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:40.912671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:40.912679 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:40.914174 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:40.914604 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:40.914605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:40.914605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:40.914668 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:40.914931 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:40.914932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:40.914932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:40.915054 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:40.915130 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:40.915419 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 07:18:40.915512 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:18:40.915839 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:40.905891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:41731 (local address 127.3.246.65:43961)
0924 07:18:40.906048 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:40.906083 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:40.906972 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:40.907196 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:40.907199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:40.907450 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:40.907631 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:40.907635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.908297 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.908298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:40.909113 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:40.909115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.909595 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.909596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:40.909784 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:40.910458 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:40.910468 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:40.912413 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:40.914271 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:40.914279 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:40.914282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:40.914510 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:40.914741 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:40.914742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:40.914742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:40.914862 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:40.915122 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:40.915129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:40.915272 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 07:18:40.915397 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 07:18:40.915558 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260924 07:18:40.916612 7829 heartbeater.cc:347] Connected to a master server at 127.3.246.126:41731
I20260924 07:18:40.916718 7829 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:40.916906 7829 heartbeater.cc:511] Master 127.3.246.126:41731 requested a full tablet report, sending...
I20260924 07:18:40.917464 7638 ts_manager.cc:194] Registered new tserver with Master: 72512beea3ff48ee88a02728179e7577 (127.3.246.65:46737)
I20260924 07:18:40.918115 7638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:43961
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:18:40.935979 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:41731
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:41.016458 7834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:41.016651 7834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:41.016673 7834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:41.017750 7834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:41.017783 7834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:41.017808 7834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:41.019200 7834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:41731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:41.019381 7834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:41.019547 7834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:41.024257 7842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:41.024288 7839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:41.024257 7840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:41.024494 7834 server_base.cc:1034] running on GCE node
I20260924 07:18:41.024719 7834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:41.024947 7834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:41.026103 7834 hybrid_clock.cc:648] HybridClock initialized: now 1790234321026089 us; error 24 us; skew 500 ppm
Sep 24 07:18:41 dist-test-slave-rc1w krb5kdc[7593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:41.028302 7834 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:18:41.028723 7834 webserver.cc:460] Webserver started at http://127.3.246.66:44375/ using document root <none> and password file <none>
I20260924 07:18:41.028944 7834 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:41.029011 7834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:41.029161 7834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:41.030251 7834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "e7e2afc10e6d4fe59ce7359b9fbade56"
format_stamp: "Formatted at 2026-09-24 07:18:41 on dist-test-slave-rc1w"
server_key: "d2293dd5425b0b77edcb668c4cf59ecd"
server_key_iv: "fa9968c0c0effa341cee07d0d9334834"
server_key_version: "encryptionkey@0"
I20260924 07:18:41.030593 7834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "e7e2afc10e6d4fe59ce7359b9fbade56"
format_stamp: "Formatted at 2026-09-24 07:18:41 on dist-test-slave-rc1w"
server_key: "d2293dd5425b0b77edcb668c4cf59ecd"
server_key_iv: "fa9968c0c0effa341cee07d0d9334834"
server_key_version: "encryptionkey@0"
I20260924 07:18:41.031844 7834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:41.032582 7849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:41.032755 7834 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:41.032853 7834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "e7e2afc10e6d4fe59ce7359b9fbade56"
format_stamp: "Formatted at 2026-09-24 07:18:41 on dist-test-slave-rc1w"
server_key: "d2293dd5425b0b77edcb668c4cf59ecd"
server_key_iv: "fa9968c0c0effa341cee07d0d9334834"
server_key_version: "encryptionkey@0"
I20260924 07:18:41.032929 7834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:41.055866 7834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:41.057966 7834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:41.058152 7834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:41.058413 7834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:41.058744 7834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:41.058801 7834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:41.058847 7834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:41.058872 7834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:41.065133 7834 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:42993
I20260924 07:18:41.065176 7962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:42993 every 8 connection(s)
I20260924 07:18:41.065514 7834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:41.070869 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 7834
I20260924 07:18:41.071007 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:41.071259 4057 external_mini_cluster.cc:1428] Setting key f80317ff6871215dc7e14ca666dfb4e7
Sep 24 07:18:41 dist-test-slave-rc1w krb5kdc[7593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:41.076485 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:41.066301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:44899 (local address 127.3.246.126:41731)
0924 07:18:41.066344 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:41.066346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:41.066549 (+ 203us) server_negotiation.cc:407] Connection header received
0924 07:18:41.067466 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:41.067467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:41.067476 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:41.067536 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:41.068216 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.068818 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.070496 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.070708 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.070774 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:41.073486 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:41.073487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:41.073487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:41.073493 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:41.075079 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:41.075491 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:41.075492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:41.075493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:41.075560 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:41.075809 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:41.075810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:41.075811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:41.075937 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:41.076017 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:41.076303 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 07:18:41.076396 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:18:41.076673 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:41.066251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:41731 (local address 127.3.246.66:44899)
0924 07:18:41.066418 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:41.066460 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:41.067376 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:41.067602 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:41.067606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:41.067858 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:41.068046 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:41.068050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.068990 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.068991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:41.070213 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:41.070227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.070820 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.070822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:41.071014 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:41.071732 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:41.071740 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:41.073381 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:41.075164 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:41.075170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:41.075172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:41.075412 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:41.075629 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:41.075629 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:41.075630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:41.075727 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:41.076021 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:41.076028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:41.076158 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 07:18:41.076271 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 07:18:41.076433 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260924 07:18:41.077162 7963 heartbeater.cc:347] Connected to a master server at 127.3.246.126:41731
I20260924 07:18:41.077282 7963 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:41.077499 7963 heartbeater.cc:511] Master 127.3.246.126:41731 requested a full tablet report, sending...
I20260924 07:18:41.077952 7638 ts_manager.cc:194] Registered new tserver with Master: e7e2afc10e6d4fe59ce7359b9fbade56 (127.3.246.66:42993)
I20260924 07:18:41.078331 7638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:44899
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:18:41.096772 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:41731
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:41.176877 7968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:41.177131 7968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:41.177172 7968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:41.178516 7968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:41.178578 7968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:41.178623 7968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:41.180104 7968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:41731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:41.180362 7968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:41.180580 7968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:41.185616 7973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:41.185859 7968 server_base.cc:1034] running on GCE node
W20260924 07:18:41.185621 7976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:41.185626 7974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:41.186241 7968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:41.186525 7968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:41.187677 7968 hybrid_clock.cc:648] HybridClock initialized: now 1790234321187655 us; error 37 us; skew 500 ppm
Sep 24 07:18:41 dist-test-slave-rc1w krb5kdc[7593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:41.190269 7968 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:18:41.190732 7968 webserver.cc:460] Webserver started at http://127.3.246.67:37513/ using document root <none> and password file <none>
I20260924 07:18:41.190949 7968 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:41.190994 7968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:41.191136 7968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:41.192157 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "f86033d4af494d2c9728d63e8b4cd51c"
format_stamp: "Formatted at 2026-09-24 07:18:41 on dist-test-slave-rc1w"
server_key: "a36c9dae30802d8e439db1e00d34d025"
server_key_iv: "789e40c8ce6e7c03c13584984ffedd2b"
server_key_version: "encryptionkey@0"
I20260924 07:18:41.192512 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "f86033d4af494d2c9728d63e8b4cd51c"
format_stamp: "Formatted at 2026-09-24 07:18:41 on dist-test-slave-rc1w"
server_key: "a36c9dae30802d8e439db1e00d34d025"
server_key_iv: "789e40c8ce6e7c03c13584984ffedd2b"
server_key_version: "encryptionkey@0"
I20260924 07:18:41.193842 7968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:41.194530 7983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:41.194720 7968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:41.194808 7968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "f86033d4af494d2c9728d63e8b4cd51c"
format_stamp: "Formatted at 2026-09-24 07:18:41 on dist-test-slave-rc1w"
server_key: "a36c9dae30802d8e439db1e00d34d025"
server_key_iv: "789e40c8ce6e7c03c13584984ffedd2b"
server_key_version: "encryptionkey@0"
I20260924 07:18:41.194878 7968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:41.205536 7968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:41.207528 7968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:41.207679 7968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:41.207856 7968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:41.208206 7968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:41.208268 7968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:41.208315 7968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:41.208343 7968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:41.214694 7968 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:46437
I20260924 07:18:41.214743 8096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:46437 every 8 connection(s)
I20260924 07:18:41.215103 7968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:41.221333 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 7968
Sep 24 07:18:41 dist-test-slave-rc1w krb5kdc[7593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:41.221470 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:41.221712 4057 external_mini_cluster.cc:1428] Setting key 8946b7841aaa07a469b79bca271efa0f
I20260924 07:18:41.224522 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:41.215878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:53075 (local address 127.3.246.126:41731)
0924 07:18:41.215937 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:41.215939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:41.216129 (+ 190us) server_negotiation.cc:407] Connection header received
0924 07:18:41.217136 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:41.217138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:41.217150 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:41.217216 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:41.217908 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.218420 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.219419 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.219624 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.219711 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:41.221982 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:41.221984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:41.221984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:41.221992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:41.223432 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:41.223790 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:41.223791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:41.223791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:41.223840 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:41.224043 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:41.224043 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:41.224043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:41.224143 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:41.224194 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:41.224403 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 07:18:41.224459 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:18:41.224766 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:41.215842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:41731 (local address 127.3.246.67:53075)
0924 07:18:41.216003 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:41.216037 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:41.217020 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:41.217288 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:41.217291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:41.217544 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:41.217813 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:41.217818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.218517 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.218518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:41.219325 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:41.219327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.219728 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.219729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:41.219848 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:41.220397 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:41.220406 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:41.221872 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:41.223507 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:41.223512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:41.223514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:41.223726 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:41.223897 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:41.223898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:41.223898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:41.223989 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:41.224196 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:41.224200 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:41.224314 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 07:18:41.224392 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:18:41.224505 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
I20260924 07:18:41.225243 8097 heartbeater.cc:347] Connected to a master server at 127.3.246.126:41731
I20260924 07:18:41.225342 8097 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:41.225510 8097 heartbeater.cc:511] Master 127.3.246.126:41731 requested a full tablet report, sending...
I20260924 07:18:41.225920 7638 ts_manager.cc:194] Registered new tserver with Master: f86033d4af494d2c9728d63e8b4cd51c (127.3.246.67:46437)
I20260924 07:18:41.226277 7638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:53075
I20260924 07:18:41.233300 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:18:41 dist-test-slave-rc1w krb5kdc[7593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:18:41 dist-test-slave-rc1w krb5kdc[7593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.65@KRBTEST.COM
I20260924 07:18:41.255839 8104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:41.248262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.3.246.65:46737)
0924 07:18:41.248476 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:41.248481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:41.248492 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:18:41.248528 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:41.248530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:41.248543 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:41.248616 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:41.249145 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.249928 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.250617 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.250927 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.250983 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:41.253110 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:41.253117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:41.253118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:41.253123 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:41.254567 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:41.254869 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:41.254873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:41.254873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:41.254981 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:41.255209 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:41.255210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:41.255210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:41.255352 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:41.255533 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:41.255608 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 07:18:41.255712 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":107,"threads_started":1}
Sep 24 07:18:41 dist-test-slave-rc1w krb5kdc[7593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:41.263550 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:41.257128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.3.246.126:41731)
0924 07:18:41.257172 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:41.257174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:41.257185 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:18:41.257254 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:41.257255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:41.257264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:41.257307 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:41.257806 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.258288 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.258921 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.259101 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.259150 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:41.261257 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:41.261258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:41.261259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:41.261265 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:41.262625 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:41.262920 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:41.262921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:41.262921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:41.262969 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:41.263127 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:41.263127 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:41.263127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:41.263260 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:41.263335 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:41.263428 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 07:18:41.263487 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 07:18:41.263820 7638 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35030
I20260924 07:18:41.264209 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 7700
I20260924 07:18:41.268553 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 7834
I20260924 07:18:41.273375 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 7968
I20260924 07:18:41.277609 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 7606
2026-09-24T07:18:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1883 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:41 dist-test-slave-rc1w krb5kdc[8109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:41 dist-test-slave-rc1w krb5kdc[8109](info): set up 2 sockets
Sep 24 07:18:41 dist-test-slave-rc1w krb5kdc[8109](info): commencing operation
krb5kdc: starting...
W20260924 07:18:42.309732 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:42 dist-test-slave-rc1w krb5kdc[8109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:18:42.428980 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37385
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:38751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:37385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:42.511252 8122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:42.511521 8122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:42.511559 8122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:42.512725 8122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:42.512794 8122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:42.512820 8122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:42.512838 8122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:42.512866 8122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:42.514377 8122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:38751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:37385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:42.514659 8122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:42.514895 8122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:42.519587 8130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:42.519590 8127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:42.519843 8122 server_base.cc:1034] running on GCE node
W20260924 07:18:42.519604 8128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:42.520119 8122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:42.520387 8122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:42.521580 8122 hybrid_clock.cc:648] HybridClock initialized: now 1790234322521589 us; error 49 us; skew 500 ppm
Sep 24 07:18:42 dist-test-slave-rc1w krb5kdc[8109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:42.524120 8122 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:18:42.524529 8122 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:42.524605 8122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:42.524741 8122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:42.525880 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "483d5e7267a14cca86c53c2d3c6dd93a"
format_stamp: "Formatted at 2026-09-24 07:18:42 on dist-test-slave-rc1w"
server_key: "3dad9d8ecc253383284dab94a38e1aaa"
server_key_iv: "e3478a66fa378fbe1c0f2aaac8ec4193"
server_key_version: "encryptionkey@0"
I20260924 07:18:42.526235 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "483d5e7267a14cca86c53c2d3c6dd93a"
format_stamp: "Formatted at 2026-09-24 07:18:42 on dist-test-slave-rc1w"
server_key: "3dad9d8ecc253383284dab94a38e1aaa"
server_key_iv: "e3478a66fa378fbe1c0f2aaac8ec4193"
server_key_version: "encryptionkey@0"
I20260924 07:18:42.527513 8122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:42.528220 8136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:42.528384 8122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:42.528482 8122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "483d5e7267a14cca86c53c2d3c6dd93a"
format_stamp: "Formatted at 2026-09-24 07:18:42 on dist-test-slave-rc1w"
server_key: "3dad9d8ecc253383284dab94a38e1aaa"
server_key_iv: "e3478a66fa378fbe1c0f2aaac8ec4193"
server_key_version: "encryptionkey@0"
I20260924 07:18:42.528564 8122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:42.534896 8122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:42.536764 8122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:42.536931 8122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:42.540961 8122 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:37385
I20260924 07:18:42.540983 8188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:37385 every 8 connection(s)
I20260924 07:18:42.541271 8122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:42.542227 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:42.543486 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 8122
I20260924 07:18:42.543567 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:42.543768 4057 external_mini_cluster.cc:1428] Setting key 1787b7a4e60f19a9026781be89a43080
I20260924 07:18:42.544927 8189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a: Bootstrap starting.
I20260924 07:18:42.545605 8189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:42.545980 8189 log.cc:824] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:42.546710 8189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a: No bootstrap required, opened a new log
I20260924 07:18:42.548941 8189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483d5e7267a14cca86c53c2d3c6dd93a" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37385 } }
I20260924 07:18:42.549098 8189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:42.549145 8189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 483d5e7267a14cca86c53c2d3c6dd93a, State: Initialized, Role: FOLLOWER
I20260924 07:18:42.549306 8189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483d5e7267a14cca86c53c2d3c6dd93a" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37385 } }
I20260924 07:18:42.549403 8189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:42.549443 8189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:42.549494 8189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:42.550107 8189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483d5e7267a14cca86c53c2d3c6dd93a" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37385 } }
I20260924 07:18:42.550235 8189 leader_election.cc:302] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 483d5e7267a14cca86c53c2d3c6dd93a; no voters:
I20260924 07:18:42.550412 8189 leader_election.cc:288] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:42.550459 8194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:42.550606 8194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [term 1 LEADER]: Becoming Leader. State: Replica: 483d5e7267a14cca86c53c2d3c6dd93a, State: Running, Role: LEADER
I20260924 07:18:42.550779 8194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483d5e7267a14cca86c53c2d3c6dd93a" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37385 } }
Sep 24 07:18:42 dist-test-slave-rc1w krb5kdc[8109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:42.550676 8189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:42.551966 8196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "483d5e7267a14cca86c53c2d3c6dd93a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483d5e7267a14cca86c53c2d3c6dd93a" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37385 } } }
I20260924 07:18:42.552064 8196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:42.552449 8195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 483d5e7267a14cca86c53c2d3c6dd93a. Latest consensus state: current_term: 1 leader_uuid: "483d5e7267a14cca86c53c2d3c6dd93a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483d5e7267a14cca86c53c2d3c6dd93a" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37385 } } }
I20260924 07:18:42.552515 8195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:42.552352 8209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:42.553151 8209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:42.554827 8209 catalog_manager.cc:1347] Generated new cluster ID: 36b1283847d2414887015b9d6ae0d2eb
I20260924 07:18:42.554883 8209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:42.555402 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:42.544286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.3.246.126:37385)
0924 07:18:42.544468 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:42.544478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:42.544515 (+ 37us) server_negotiation.cc:407] Connection header received
0924 07:18:42.545056 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:42.545065 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:42.545122 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:42.545284 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:42.545919 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.547244 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.547996 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.548697 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.548817 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:42.551467 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:42.551472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:42.551472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:42.551484 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:42.553593 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:42.554039 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:42.554042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:42.554042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:42.554141 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:42.554390 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:42.554391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:42.554392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:42.554693 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:42.554830 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:42.555070 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 07:18:42.555197 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260924 07:18:42.563402 8209 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:42.564136 8209 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:42.568625 8209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a: Generated new TSK 0
I20260924 07:18:42.568809 8209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:18:42.597249 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37385
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:38751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:42.678267 8215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:42.678467 8215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:42.678489 8215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:42.679603 8215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:42.679641 8215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:42.679668 8215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:42.681090 8215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:38751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:42.681281 8215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:42.681452 8215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:42.686556 8221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:42.686573 8223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:42.686673 8215 server_base.cc:1034] running on GCE node
W20260924 07:18:42.686643 8220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:42.686951 8215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:42.687156 8215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:42.688303 8215 hybrid_clock.cc:648] HybridClock initialized: now 1790234322688278 us; error 34 us; skew 500 ppm
Sep 24 07:18:42 dist-test-slave-rc1w krb5kdc[8109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:42.690856 8215 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:18:42.691227 8215 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:42.691309 8215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:42.691468 8215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:42.692507 8215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "063e998589d645109c359e9a907b701b"
format_stamp: "Formatted at 2026-09-24 07:18:42 on dist-test-slave-rc1w"
server_key: "b961506d6290620d2a83452119c6e02f"
server_key_iv: "aba1410a19aa50f9a0255062117e1c01"
server_key_version: "encryptionkey@0"
I20260924 07:18:42.692862 8215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "063e998589d645109c359e9a907b701b"
format_stamp: "Formatted at 2026-09-24 07:18:42 on dist-test-slave-rc1w"
server_key: "b961506d6290620d2a83452119c6e02f"
server_key_iv: "aba1410a19aa50f9a0255062117e1c01"
server_key_version: "encryptionkey@0"
I20260924 07:18:42.694278 8215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:18:42.695073 8229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:42.695308 8215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:42.695442 8215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "063e998589d645109c359e9a907b701b"
format_stamp: "Formatted at 2026-09-24 07:18:42 on dist-test-slave-rc1w"
server_key: "b961506d6290620d2a83452119c6e02f"
server_key_iv: "aba1410a19aa50f9a0255062117e1c01"
server_key_version: "encryptionkey@0"
I20260924 07:18:42.695518 8215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:42.709627 8215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:42.711645 8215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:42.711812 8215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:42.712038 8215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:42.712343 8215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:42.712399 8215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:42.712440 8215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:42.712468 8215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:42.719327 8215 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:34913
I20260924 07:18:42.719410 8342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:34913 every 8 connection(s)
I20260924 07:18:42.719661 8215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:42.722416 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 8215
I20260924 07:18:42.722530 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:42.722707 4057 external_mini_cluster.cc:1428] Setting key 934b7a4748ba482700a96f0b33ecca05
Sep 24 07:18:42 dist-test-slave-rc1w krb5kdc[8109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:42.730103 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:42.720412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:46497 (local address 127.3.246.126:37385)
0924 07:18:42.720490 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:42.720492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:42.720642 (+ 150us) server_negotiation.cc:407] Connection header received
0924 07:18:42.721120 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:42.721122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:42.721132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:42.721191 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:42.722165 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.722969 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.724098 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.724447 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.724517 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:42.726951 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:42.726953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:42.726953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:42.726961 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:42.728650 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:42.729064 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:42.729066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:42.729066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:42.729131 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:42.729375 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:42.729376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:42.729376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:42.729524 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:42.729597 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:42.729906 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 07:18:42.729993 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:18:42.730134 8345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:42.720360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37385 (local address 127.3.246.65:46497)
0924 07:18:42.720514 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:42.720556 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:42.721026 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:42.721244 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:42.721247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:42.721490 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:42.722071 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:42.722076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.723075 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.723076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:42.723987 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:42.723989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.724536 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.724538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:42.724695 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:42.725230 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:42.725241 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:42.726842 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:42.728773 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:42.728779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:42.728781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:42.728987 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:42.729211 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:42.729211 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:42.729212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:42.729303 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:42.729592 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:42.729596 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:42.729756 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 07:18:42.729856 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 07:18:42.729976 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260924 07:18:42.730645 8343 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37385
I20260924 07:18:42.730760 8343 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:42.730913 8343 heartbeater.cc:511] Master 127.3.246.126:37385 requested a full tablet report, sending...
I20260924 07:18:42.731501 8153 ts_manager.cc:194] Registered new tserver with Master: 063e998589d645109c359e9a907b701b (127.3.246.65:34913)
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
I20260924 07:18:42.732406 8153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:46497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:18:42.749388 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37385
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:38751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:42.831861 8348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:42.832059 8348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:42.832083 8348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:42.833276 8348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:42.833320 8348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:42.833348 8348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:42.834893 8348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:38751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:42.835142 8348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:42.835388 8348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:42.840209 8356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:42.840209 8354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:42.840240 8353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:42.840286 8348 server_base.cc:1034] running on GCE node
I20260924 07:18:42.840616 8348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:42.840801 8348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:42.841959 8348 hybrid_clock.cc:648] HybridClock initialized: now 1790234322841944 us; error 31 us; skew 500 ppm
Sep 24 07:18:42 dist-test-slave-rc1w krb5kdc[8109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:42.844375 8348 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:18:42.844688 8348 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:42.844733 8348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:42.844892 8348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:42.845966 8348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "f1a3a002893445569a0a3c7c5c4d4e39"
format_stamp: "Formatted at 2026-09-24 07:18:42 on dist-test-slave-rc1w"
server_key: "fddc3536c337eef95cd48304830abb06"
server_key_iv: "aaee5bed4b0da3ae1c78ecf6ad33814c"
server_key_version: "encryptionkey@0"
I20260924 07:18:42.846302 8348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "f1a3a002893445569a0a3c7c5c4d4e39"
format_stamp: "Formatted at 2026-09-24 07:18:42 on dist-test-slave-rc1w"
server_key: "fddc3536c337eef95cd48304830abb06"
server_key_iv: "aaee5bed4b0da3ae1c78ecf6ad33814c"
server_key_version: "encryptionkey@0"
I20260924 07:18:42.847512 8348 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:42.848248 8362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:42.848448 8348 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:42.848542 8348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "f1a3a002893445569a0a3c7c5c4d4e39"
format_stamp: "Formatted at 2026-09-24 07:18:42 on dist-test-slave-rc1w"
server_key: "fddc3536c337eef95cd48304830abb06"
server_key_iv: "aaee5bed4b0da3ae1c78ecf6ad33814c"
server_key_version: "encryptionkey@0"
I20260924 07:18:42.848619 8348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:42.877147 8348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:42.879207 8348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:42.879370 8348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:42.879604 8348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:42.879910 8348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:42.879968 8348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:42.880012 8348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:42.880036 8348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:42.885876 8348 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:41831
I20260924 07:18:42.885922 8475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:41831 every 8 connection(s)
I20260924 07:18:42.886176 8348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:18:42 dist-test-slave-rc1w krb5kdc[8109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:42.894479 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 8348
I20260924 07:18:42.894585 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:42.894783 4057 external_mini_cluster.cc:1428] Setting key d7f61f1ce91dc4d376fea92ea920912c
I20260924 07:18:42.895401 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:42.886920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:47895 (local address 127.3.246.126:37385)
0924 07:18:42.886967 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:42.886969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:42.887184 (+ 215us) server_negotiation.cc:407] Connection header received
0924 07:18:42.887703 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:42.887704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:42.887716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:42.887771 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:42.888713 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.889209 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.890296 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.890496 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.890582 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:42.892670 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:42.892672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:42.892672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:42.892679 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:42.893911 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:42.894253 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:42.894254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:42.894254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:42.894303 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:42.894625 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:42.894626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:42.894627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:42.894764 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:42.894850 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:42.895212 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 07:18:42.895297 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:18:42.895454 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:42.886895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37385 (local address 127.3.246.66:47895)
0924 07:18:42.887060 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:42.887095 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:42.887595 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:42.887846 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:42.887850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:42.888108 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:42.888599 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:42.888604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.889300 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.889301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:42.890201 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:42.890204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.890593 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.890594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:42.890717 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:42.891185 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:42.891192 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:42.892551 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:42.893982 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:42.893988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:42.893989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:42.894190 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:42.894375 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:42.894376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:42.894377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:42.894518 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:42.894839 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:42.894847 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:42.895000 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 07:18:42.895124 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 07:18:42.895255 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260924 07:18:42.896041 8476 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37385
I20260924 07:18:42.896140 8476 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:42.896293 8476 heartbeater.cc:511] Master 127.3.246.126:37385 requested a full tablet report, sending...
I20260924 07:18:42.896620 8153 ts_manager.cc:194] Registered new tserver with Master: f1a3a002893445569a0a3c7c5c4d4e39 (127.3.246.66:41831)
I20260924 07:18:42.897091 8153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:47895
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:18:42.922554 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37385
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:38751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:43.002991 8481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:43.003183 8481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:43.003206 8481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:43.004266 8481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:43.004299 8481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:43.004325 8481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:43.005739 8481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:38751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:43.005914 8481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:43.006079 8481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:43.011031 8487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:43.011032 8486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:43.011032 8489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:43.011186 8481 server_base.cc:1034] running on GCE node
I20260924 07:18:43.011492 8481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:43.011747 8481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:43.012928 8481 hybrid_clock.cc:648] HybridClock initialized: now 1790234323012900 us; error 38 us; skew 500 ppm
Sep 24 07:18:43 dist-test-slave-rc1w krb5kdc[8109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234323, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:43.015467 8481 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:18:43.015769 8481 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:43.015812 8481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:43.015900 8481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:43.016860 8481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "740edeb8f3084486bdb6510c812b889c"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "07623e5fd5a8b841258d9d7cc28268d9"
server_key_iv: "7db823cdce259c8a0a0f9402518161ed"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.017150 8481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "740edeb8f3084486bdb6510c812b889c"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "07623e5fd5a8b841258d9d7cc28268d9"
server_key_iv: "7db823cdce259c8a0a0f9402518161ed"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.018399 8481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:43.019052 8495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:43.019235 8481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:18:43.019311 8481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "740edeb8f3084486bdb6510c812b889c"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "07623e5fd5a8b841258d9d7cc28268d9"
server_key_iv: "7db823cdce259c8a0a0f9402518161ed"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.019371 8481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:43.044720 8481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:43.046912 8481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:43.047077 8481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:43.047269 8481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:43.047583 8481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:43.047614 8481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:43.047637 8481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:43.047686 8481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:43.053550 8481 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:43931
I20260924 07:18:43.053638 8608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:43931 every 8 connection(s)
I20260924 07:18:43.053939 8481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:43.057258 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 8481
I20260924 07:18:43.057359 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:43.057523 4057 external_mini_cluster.cc:1428] Setting key 2d481475ff82926b0fa7b756e8a842f3
Sep 24 07:18:43 dist-test-slave-rc1w krb5kdc[8109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234323, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:43.062979 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.054698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:35483 (local address 127.3.246.126:37385)
0924 07:18:43.054767 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.054769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.054942 (+ 173us) server_negotiation.cc:407] Connection header received
0924 07:18:43.055441 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.055442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.055458 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.055543 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:43.056429 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.056913 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.058253 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.058462 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.058548 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.060599 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:43.060601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:43.060601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:43.060608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:43.061858 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:43.062201 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:43.062202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:43.062202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:43.062248 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:43.062465 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:43.062465 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:43.062466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:43.062565 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:43.062612 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.062848 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.062917 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:18:43.063138 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.054667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37385 (local address 127.3.246.67:35483)
0924 07:18:43.054822 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.054863 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.055326 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.055609 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.055612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.055861 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:43.056350 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.056354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.056990 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.056991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.058141 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.058144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.058558 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.058559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.058693 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.059119 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:43.059127 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:43.060510 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:43.061926 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:43.061932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:43.061933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:43.062139 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:43.062317 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:43.062317 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:43.062317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:43.062410 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:43.062619 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:43.062623 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:43.062728 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.062809 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.062926 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260924 07:18:43.063616 8609 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37385
I20260924 07:18:43.063709 8609 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:43.063815 8609 heartbeater.cc:511] Master 127.3.246.126:37385 requested a full tablet report, sending...
I20260924 07:18:43.064157 8153 ts_manager.cc:194] Registered new tserver with Master: 740edeb8f3084486bdb6510c812b889c (127.3.246.67:43931)
I20260924 07:18:43.064532 8153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:35483
I20260924 07:18:43.069236 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:18:43.076289 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.070533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50160 (local address 127.3.246.126:37385)
0924 07:18:43.070588 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.070590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.070710 (+ 120us) server_negotiation.cc:407] Connection header received
0924 07:18:43.070794 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.070794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.070807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.070873 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:43.071408 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.071875 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.072597 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.072783 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.072864 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.074021 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:43.074023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:43.074023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:43.074029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:43.075124 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:43.075470 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:43.075472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:43.075472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:43.075531 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:43.075729 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:43.075730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:43.075731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:43.075879 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:43.075990 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.076098 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.076174 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:18:43.077652 8153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:43.079191 8153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:43.085755 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.081332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:43931 (local address 127.0.0.1:40964)
0924 07:18:43.081752 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.081761 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.081807 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.082031 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.082033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.082036 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:43.082104 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.082105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.083440 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.083442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.084383 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.084385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.085441 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.085442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.085530 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.085534 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.085599 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.085661 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260924 07:18:43.085803 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.081854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.3.246.66:41831)
0924 07:18:43.082144 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.082148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.082158 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:18:43.082194 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.082196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.082212 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.082283 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:43.082452 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.083175 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.084694 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.085168 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.085339 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.085409 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.085650 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.085717 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":86,"threads_started":1}
I20260924 07:18:43.085959 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.081453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.3.246.65:34913)
0924 07:18:43.081646 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.081652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.082005 (+ 353us) server_negotiation.cc:407] Connection header received
0924 07:18:43.082048 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.082050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.082064 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.082282 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:43.082362 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.083385 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.084657 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.085473 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.085539 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.085590 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.085702 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.085743 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260924 07:18:43.086007 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.081613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:41831 (local address 127.0.0.1:59172)
0924 07:18:43.081855 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.081866 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.081920 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.082340 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.082341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.082343 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:43.082386 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.082388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.083290 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.083291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.084609 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.084612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.085442 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.085442 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.085503 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.085534 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.085603 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.085951 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":295,"thread_start_us":91,"threads_started":1}
I20260924 07:18:43.086015 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.081495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40964 (local address 127.3.246.67:43931)
0924 07:18:43.081675 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.081679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.081888 (+ 209us) server_negotiation.cc:407] Connection header received
0924 07:18:43.081927 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.081929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.081945 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.082021 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:43.082190 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.083328 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.084466 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.085163 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.085345 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.085410 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.085881 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.085928 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
I20260924 07:18:43.086282 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.081332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:34913 (local address 127.0.0.1:38618)
0924 07:18:43.081648 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.081717 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.081801 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.082165 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.082165 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.082168 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:43.082198 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.082199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.083475 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.083476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.084572 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.084575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.085552 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.085553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.085615 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.085619 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.085706 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.085745 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":35,"thread_start_us":60,"threads_started":1}
I20260924 07:18:43.086685 8543 tablet_service.cc:1467] Processing CreateTablet for tablet 6d84c98b3f7f47a0a451851e35f87530 (DEFAULT_TABLE table=test-table [id=4beae74d066c4719acd0a83978ef72a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:43.086704 8410 tablet_service.cc:1467] Processing CreateTablet for tablet 6d84c98b3f7f47a0a451851e35f87530 (DEFAULT_TABLE table=test-table [id=4beae74d066c4719acd0a83978ef72a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:43.087045 8543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d84c98b3f7f47a0a451851e35f87530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:43.087045 8410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d84c98b3f7f47a0a451851e35f87530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:43.087052 8277 tablet_service.cc:1467] Processing CreateTablet for tablet 6d84c98b3f7f47a0a451851e35f87530 (DEFAULT_TABLE table=test-table [id=4beae74d066c4719acd0a83978ef72a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:43.087324 8277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d84c98b3f7f47a0a451851e35f87530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:43.089502 8626 tablet_bootstrap.cc:492] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c: Bootstrap starting.
I20260924 07:18:43.090134 8626 tablet_bootstrap.cc:654] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:43.090417 8628 tablet_bootstrap.cc:492] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b: Bootstrap starting.
I20260924 07:18:43.090994 8626 log.cc:824] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:43.091102 8628 tablet_bootstrap.cc:654] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:43.091696 8628 log.cc:824] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:43.091754 8626 tablet_bootstrap.cc:492] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c: No bootstrap required, opened a new log
I20260924 07:18:43.091811 8626 ts_tablet_manager.cc:1397] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:18:43.092396 8628 tablet_bootstrap.cc:492] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b: No bootstrap required, opened a new log
I20260924 07:18:43.092464 8628 ts_tablet_manager.cc:1397] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:18:43.093820 8626 raft_consensus.cc:348] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063e998589d645109c359e9a907b701b" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34913 } } peers { permanent_uuid: "f1a3a002893445569a0a3c7c5c4d4e39" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41831 } } peers { permanent_uuid: "740edeb8f3084486bdb6510c812b889c" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43931 } }
I20260924 07:18:43.093989 8626 raft_consensus.cc:374] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:43.094059 8626 raft_consensus.cc:729] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 740edeb8f3084486bdb6510c812b889c, State: Initialized, Role: FOLLOWER
I20260924 07:18:43.094188 8626 consensus_queue.cc:260] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063e998589d645109c359e9a907b701b" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34913 } } peers { permanent_uuid: "f1a3a002893445569a0a3c7c5c4d4e39" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41831 } } peers { permanent_uuid: "740edeb8f3084486bdb6510c812b889c" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43931 } }
I20260924 07:18:43.094475 8626 ts_tablet_manager.cc:1428] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 07:18:43.094491 8609 heartbeater.cc:503] Master 127.3.246.126:37385 was elected leader, sending a full tablet report...
I20260924 07:18:43.094441 8628 raft_consensus.cc:348] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063e998589d645109c359e9a907b701b" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34913 } } peers { permanent_uuid: "f1a3a002893445569a0a3c7c5c4d4e39" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41831 } } peers { permanent_uuid: "740edeb8f3084486bdb6510c812b889c" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43931 } }
I20260924 07:18:43.091361 8627 tablet_bootstrap.cc:492] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39: Bootstrap starting.
I20260924 07:18:43.094594 8628 raft_consensus.cc:374] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:43.094672 8628 raft_consensus.cc:729] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063e998589d645109c359e9a907b701b, State: Initialized, Role: FOLLOWER
I20260924 07:18:43.094849 8628 consensus_queue.cc:260] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063e998589d645109c359e9a907b701b" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34913 } } peers { permanent_uuid: "f1a3a002893445569a0a3c7c5c4d4e39" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41831 } } peers { permanent_uuid: "740edeb8f3084486bdb6510c812b889c" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43931 } }
I20260924 07:18:43.095130 8343 heartbeater.cc:503] Master 127.3.246.126:37385 was elected leader, sending a full tablet report...
I20260924 07:18:43.095186 8627 tablet_bootstrap.cc:654] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:43.095129 8628 ts_tablet_manager.cc:1428] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:43.095827 8627 log.cc:824] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:43.096592 8627 tablet_bootstrap.cc:492] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39: No bootstrap required, opened a new log
I20260924 07:18:43.096643 8627 ts_tablet_manager.cc:1397] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 07:18:43.098259 8627 raft_consensus.cc:348] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063e998589d645109c359e9a907b701b" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34913 } } peers { permanent_uuid: "f1a3a002893445569a0a3c7c5c4d4e39" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41831 } } peers { permanent_uuid: "740edeb8f3084486bdb6510c812b889c" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43931 } }
I20260924 07:18:43.098394 8627 raft_consensus.cc:374] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:43.098416 8627 raft_consensus.cc:729] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a3a002893445569a0a3c7c5c4d4e39, State: Initialized, Role: FOLLOWER
I20260924 07:18:43.098489 8627 consensus_queue.cc:260] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063e998589d645109c359e9a907b701b" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34913 } } peers { permanent_uuid: "f1a3a002893445569a0a3c7c5c4d4e39" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41831 } } peers { permanent_uuid: "740edeb8f3084486bdb6510c812b889c" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43931 } }
I20260924 07:18:43.098675 8627 ts_tablet_manager.cc:1428] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:18:43.098763 8476 heartbeater.cc:503] Master 127.3.246.126:37385 was elected leader, sending a full tablet report...
W20260924 07:18:43.136771 8477 tablet.cc:2369] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:18:43.220376 8344 tablet.cc:2369] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:18:43.304522 8610 tablet.cc:2369] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:18:43.324791 8632 raft_consensus.cc:482] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:43.324985 8632 raft_consensus.cc:504] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063e998589d645109c359e9a907b701b" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34913 } } peers { permanent_uuid: "f1a3a002893445569a0a3c7c5c4d4e39" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41831 } } peers { permanent_uuid: "740edeb8f3084486bdb6510c812b889c" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43931 } }
I20260924 07:18:43.325383 8632 leader_election.cc:288] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a3a002893445569a0a3c7c5c4d4e39 (127.3.246.66:41831), 740edeb8f3084486bdb6510c812b889c (127.3.246.67:43931)
I20260924 07:18:43.329190 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.325876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:44627 (local address 127.3.246.67:43931)
0924 07:18:43.325928 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.325930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.325968 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:18:43.326088 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.326090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.326102 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.326169 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:43.326416 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.327006 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.328442 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.328837 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.328924 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.328939 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.329085 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.329122 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:18:43.329200 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.325781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:43931 (local address 127.3.246.65:44627)
0924 07:18:43.325907 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.325918 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.325984 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.326213 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.326214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.326217 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:43.326314 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.326318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.327077 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.327078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.328352 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.328355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.328923 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.328924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.328999 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.329002 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.329066 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.329107 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260924 07:18:43.329605 8563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d84c98b3f7f47a0a451851e35f87530" candidate_uuid: "063e998589d645109c359e9a907b701b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "740edeb8f3084486bdb6510c812b889c" is_pre_election: true
I20260924 07:18:43.329818 8563 raft_consensus.cc:2454] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 063e998589d645109c359e9a907b701b in term 0.
I20260924 07:18:43.329882 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.326111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:41831 (local address 127.3.246.65:52291)
0924 07:18:43.326268 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.326281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.326353 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.327242 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.327242 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.327244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:43.327299 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.327300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.328136 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.328137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.328873 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.328874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.329586 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.329587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.329657 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.329662 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.329757 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.329798 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
I20260924 07:18:43.329887 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.325738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:52291 (local address 127.3.246.66:41831)
0924 07:18:43.325788 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.325790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.326352 (+ 562us) server_negotiation.cc:407] Connection header received
0924 07:18:43.327151 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.327152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.327164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.327228 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:43.327366 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.328055 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.328973 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.329492 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.329571 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.329580 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.329764 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.329808 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:18:43.330058 8232 leader_election.cc:302] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063e998589d645109c359e9a907b701b, 740edeb8f3084486bdb6510c812b889c; no voters:
I20260924 07:18:43.330204 8632 raft_consensus.cc:2781] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:43.330262 8632 raft_consensus.cc:482] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:43.330296 8632 raft_consensus.cc:3037] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:43.330341 8430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d84c98b3f7f47a0a451851e35f87530" candidate_uuid: "063e998589d645109c359e9a907b701b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a3a002893445569a0a3c7c5c4d4e39" is_pre_election: true
I20260924 07:18:43.330506 8430 raft_consensus.cc:2454] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 063e998589d645109c359e9a907b701b in term 0.
I20260924 07:18:43.331030 8632 raft_consensus.cc:504] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063e998589d645109c359e9a907b701b" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34913 } } peers { permanent_uuid: "f1a3a002893445569a0a3c7c5c4d4e39" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41831 } } peers { permanent_uuid: "740edeb8f3084486bdb6510c812b889c" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43931 } }
I20260924 07:18:43.331176 8632 leader_election.cc:288] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [CANDIDATE]: Term 1 election: Requested vote from peers f1a3a002893445569a0a3c7c5c4d4e39 (127.3.246.66:41831), 740edeb8f3084486bdb6510c812b889c (127.3.246.67:43931)
I20260924 07:18:43.331322 8430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d84c98b3f7f47a0a451851e35f87530" candidate_uuid: "063e998589d645109c359e9a907b701b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a3a002893445569a0a3c7c5c4d4e39"
I20260924 07:18:43.331327 8563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d84c98b3f7f47a0a451851e35f87530" candidate_uuid: "063e998589d645109c359e9a907b701b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "740edeb8f3084486bdb6510c812b889c"
I20260924 07:18:43.331408 8563 raft_consensus.cc:3037] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:43.331429 8430 raft_consensus.cc:3037] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:43.332165 8430 raft_consensus.cc:2454] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 063e998589d645109c359e9a907b701b in term 1.
I20260924 07:18:43.332228 8563 raft_consensus.cc:2454] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 063e998589d645109c359e9a907b701b in term 1.
I20260924 07:18:43.332420 8231 leader_election.cc:302] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063e998589d645109c359e9a907b701b, f1a3a002893445569a0a3c7c5c4d4e39; no voters:
I20260924 07:18:43.332508 8632 raft_consensus.cc:2781] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:43.332692 8632 raft_consensus.cc:686] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [term 1 LEADER]: Becoming Leader. State: Replica: 063e998589d645109c359e9a907b701b, State: Running, Role: LEADER
I20260924 07:18:43.332792 8632 consensus_queue.cc:237] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063e998589d645109c359e9a907b701b" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34913 } } peers { permanent_uuid: "f1a3a002893445569a0a3c7c5c4d4e39" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41831 } } peers { permanent_uuid: "740edeb8f3084486bdb6510c812b889c" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43931 } }
I20260924 07:18:43.333659 8153 catalog_manager.cc:5495] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b reported cstate change: term changed from 0 to 1, leader changed from <none> to 063e998589d645109c359e9a907b701b (127.3.246.65). New cstate: current_term: 1 leader_uuid: "063e998589d645109c359e9a907b701b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063e998589d645109c359e9a907b701b" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a3a002893445569a0a3c7c5c4d4e39" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "740edeb8f3084486bdb6510c812b889c" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43931 } health_report { overall_health: UNKNOWN } } }
I20260924 07:18:43.369422 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.366964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.3.246.65:34913)
0924 07:18:43.367041 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.367044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.367053 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:43.367096 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.367097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.367116 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.367203 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:43.367358 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.367850 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.368622 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.368837 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.368917 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.369085 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:43.369195 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:43.369295 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.369314 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.369333 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:18:43.371443 8563 raft_consensus.cc:1261] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c [term 1 FOLLOWER]: Refusing update from remote peer 063e998589d645109c359e9a907b701b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:43.371441 8430 raft_consensus.cc:1261] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39 [term 1 FOLLOWER]: Refusing update from remote peer 063e998589d645109c359e9a907b701b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:43.371845 8632 consensus_queue.cc:1035] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [LEADER]: Connected to new peer: Peer: permanent_uuid: "740edeb8f3084486bdb6510c812b889c" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 43931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:43.372005 8637 consensus_queue.cc:1035] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a3a002893445569a0a3c7c5c4d4e39" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:43.374298 8640 mvcc.cc:205] Tried to move back new op lower bound from 7332799788526338048 to 7332799788371849216. Current Snapshot: MvccSnapshot[applied={T|T < 7332799788526338048}]
I20260924 07:18:43.374737 8642 mvcc.cc:205] Tried to move back new op lower bound from 7332799788526338048 to 7332799788371849216. Current Snapshot: MvccSnapshot[applied={T|T < 7332799788526338048}]
I20260924 07:18:43.377348 8153 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:43.377439 8153 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:43.378239 8153 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 483d5e7267a14cca86c53c2d3c6dd93a: Sending DeleteTablet for 3 replicas of tablet 6d84c98b3f7f47a0a451851e35f87530
I20260924 07:18:43.378579 8277 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d84c98b3f7f47a0a451851e35f87530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:43 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:38618
I20260924 07:18:43.378657 8543 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d84c98b3f7f47a0a451851e35f87530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:43 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:40964
I20260924 07:18:43.378650 8410 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d84c98b3f7f47a0a451851e35f87530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:43 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:59172
I20260924 07:18:43.378925 8649 tablet_replica.cc:330] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b: stopping tablet replica
I20260924 07:18:43.379026 8649 raft_consensus.cc:2229] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:43.379151 8649 raft_consensus.cc:2258] T 6d84c98b3f7f47a0a451851e35f87530 P 063e998589d645109c359e9a907b701b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:43.379237 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 8215
W20260924 07:18:43.383957 8138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:34913 (error 108)
I20260924 07:18:43.384553 8650 tablet_replica.cc:330] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c: stopping tablet replica
I20260924 07:18:43.384671 8651 tablet_replica.cc:330] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39: stopping tablet replica
I20260924 07:18:43.384718 8650 raft_consensus.cc:2229] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:43.384822 8651 raft_consensus.cc:2229] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:43.384836 8650 raft_consensus.cc:2258] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:43.384948 8651 raft_consensus.cc:2258] T 6d84c98b3f7f47a0a451851e35f87530 P f1a3a002893445569a0a3c7c5c4d4e39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:43.385025 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.384903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:34913 (local address 127.0.0.1:38642)
0924 07:18:43.384950 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.384985 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:34913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 07:18:43.385105 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 8348
I20260924 07:18:43.385342 8650 ts_tablet_manager.cc:1905] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:18:43.385747 8138 catalog_manager.cc:4584] TS 063e998589d645109c359e9a907b701b (127.3.246.65:34913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d84c98b3f7f47a0a451851e35f87530: Network error: Client connection negotiation failed: client connection to 127.3.246.65:34913: connect: Connection refused (error 111)
I20260924 07:18:43.387725 8650 ts_tablet_manager.cc:1918] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:43.387814 8650 log.cc:1196] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/6d84c98b3f7f47a0a451851e35f87530
I20260924 07:18:43.388058 8650 ts_tablet_manager.cc:1939] T 6d84c98b3f7f47a0a451851e35f87530 P 740edeb8f3084486bdb6510c812b889c: Deleting consensus metadata
I20260924 07:18:43.388506 8139 catalog_manager.cc:4841] TS 740edeb8f3084486bdb6510c812b889c (127.3.246.67:43931): tablet 6d84c98b3f7f47a0a451851e35f87530 (table test-table [id=4beae74d066c4719acd0a83978ef72a3]) successfully deleted
I20260924 07:18:43.390128 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 8481
I20260924 07:18:43.390264 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.390124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:41831 (local address 127.0.0.1:59184)
0924 07:18:43.390191 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.390227 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.66:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:18:43.394629 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 8122
2026-09-24T07:18:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2117 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T07:18:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:43Z Disabled control of system clock
I20260924 07:18:43.421984 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:42691
--webserver_interface=127.3.246.126
--webserver_port=0
--builtin_ntp_servers=127.3.246.84:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:42691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:18:43.501698 8656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:43.501904 8656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:43.501925 8656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:43.502979 8656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:43.503010 8656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:43.503023 8656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:43.503036 8656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:43.503047 8656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:43.504288 8656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:42691
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:42691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:43.504475 8656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:43.504693 8656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:43.509415 8664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:43.509426 8662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:43.509455 8656 server_base.cc:1034] running on GCE node
W20260924 07:18:43.509438 8661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:43.509970 8656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:43.510269 8656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:43.511438 8656 hybrid_clock.cc:648] HybridClock initialized: now 1790234323511426 us; error 28 us; skew 500 ppm
I20260924 07:18:43.512604 8656 webserver.cc:460] Webserver started at http://127.3.246.126:45927/ using document root <none> and password file <none>
I20260924 07:18:43.512869 8656 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:43.512940 8656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:43.513091 8656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:43.514240 8656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "9d2ff93b07e84a82bf0910c37ca1dddf"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "a370a6cbc6b3718117205e518cfd7657"
server_key_iv: "e70628549a8709bdb2cc9b76277eee54"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.514595 8656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "9d2ff93b07e84a82bf0910c37ca1dddf"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "a370a6cbc6b3718117205e518cfd7657"
server_key_iv: "e70628549a8709bdb2cc9b76277eee54"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.515888 8656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:43.516574 8670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:43.516742 8656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:43.516866 8656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "9d2ff93b07e84a82bf0910c37ca1dddf"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "a370a6cbc6b3718117205e518cfd7657"
server_key_iv: "e70628549a8709bdb2cc9b76277eee54"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.516943 8656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:43.520298 8656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:43.520556 8656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:43.520704 8656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:43.524647 8656 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:42691
I20260924 07:18:43.524693 8722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:42691 every 8 connection(s)
I20260924 07:18:43.525053 8656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:43.525751 8723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:43.525754 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 8656
I20260924 07:18:43.525876 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:43.526054 4057 external_mini_cluster.cc:1428] Setting key 895a8ce1ec995bab3d0a747ba6d75c7d
I20260924 07:18:43.528795 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.526578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.3.246.126:42691)
0924 07:18:43.526789 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.526797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.526821 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:18:43.527460 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.527467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.527507 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.527656 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:43.528359 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:43.528362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:43.528363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:43.528366 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:43.528414 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:43.528424 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:43.528501 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.528569 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.528645 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260924 07:18:43.529549 8723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf: Bootstrap starting.
I20260924 07:18:43.530169 8723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:43.530459 8723 log.cc:824] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:43.531113 8723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf: No bootstrap required, opened a new log
I20260924 07:18:43.532430 8723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2ff93b07e84a82bf0910c37ca1dddf" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42691 } }
I20260924 07:18:43.532588 8723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:43.532636 8723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2ff93b07e84a82bf0910c37ca1dddf, State: Initialized, Role: FOLLOWER
I20260924 07:18:43.532773 8723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2ff93b07e84a82bf0910c37ca1dddf" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42691 } }
I20260924 07:18:43.532864 8723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:43.532907 8723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:43.532954 8723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:43.533455 8723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2ff93b07e84a82bf0910c37ca1dddf" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42691 } }
I20260924 07:18:43.533572 8723 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d2ff93b07e84a82bf0910c37ca1dddf; no voters:
I20260924 07:18:43.533754 8723 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:43.534054 8723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:43.534063 8728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:43.534281 8728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [term 1 LEADER]: Becoming Leader. State: Replica: 9d2ff93b07e84a82bf0910c37ca1dddf, State: Running, Role: LEADER
I20260924 07:18:43.534439 8728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2ff93b07e84a82bf0910c37ca1dddf" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42691 } }
I20260924 07:18:43.534879 8729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d2ff93b07e84a82bf0910c37ca1dddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2ff93b07e84a82bf0910c37ca1dddf" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42691 } } }
I20260924 07:18:43.534886 8730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2ff93b07e84a82bf0910c37ca1dddf. Latest consensus state: current_term: 1 leader_uuid: "9d2ff93b07e84a82bf0910c37ca1dddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2ff93b07e84a82bf0910c37ca1dddf" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42691 } } }
I20260924 07:18:43.534994 8729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:43.535001 8730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf [sys.catalog]: This master's current role is: LEADER
W20260924 07:18:43.535905 8743 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:18:43.535964 8743 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:18:43.536016 8738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:43.536139 8738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:43.537807 8738 catalog_manager.cc:1347] Generated new cluster ID: 567dc05397e84a5a90d32882c318a899
I20260924 07:18:43.537905 8738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:43.553550 8738 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:43.553767 8738 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:43.559348 8738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf: Generated new TSK 0
I20260924 07:18:43.559545 8738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:18:43.561929 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42691
--builtin_ntp_servers=127.3.246.84:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:18:43.655769 8747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:43.656018 8747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:43.656055 8747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:43.657313 8747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:43.657369 8747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:43.657413 8747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:43.658831 8747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:43.659099 8747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:43.659391 8747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:43.665190 8753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:43.665190 8752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:43.665369 8747 server_base.cc:1034] running on GCE node
W20260924 07:18:43.665402 8755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:43.665612 8747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:43.665944 8747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:43.667126 8747 hybrid_clock.cc:648] HybridClock initialized: now 1790234323667103 us; error 38 us; skew 500 ppm
I20260924 07:18:43.668540 8747 webserver.cc:460] Webserver started at http://127.3.246.65:43785/ using document root <none> and password file <none>
I20260924 07:18:43.668788 8747 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:43.668857 8747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:43.669010 8747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:43.670161 8747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "2a0788878b7f4d49b95817eeec81f863"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "fe9bf753a18e7efa4b7e165d78ecf9de"
server_key_iv: "52701e979e3ed27480c28ae126b5209c"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.670524 8747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "2a0788878b7f4d49b95817eeec81f863"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "fe9bf753a18e7efa4b7e165d78ecf9de"
server_key_iv: "52701e979e3ed27480c28ae126b5209c"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.671835 8747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 07:18:43.672587 8761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:43.672947 8747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:43.673056 8747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "2a0788878b7f4d49b95817eeec81f863"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "fe9bf753a18e7efa4b7e165d78ecf9de"
server_key_iv: "52701e979e3ed27480c28ae126b5209c"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.673131 8747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:43.676476 8747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:43.676791 8747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:43.676931 8747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:43.677179 8747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:43.677547 8747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:43.677582 8747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:43.677601 8747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:43.677644 8747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:43.683444 8747 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:46329
I20260924 07:18:43.683507 8874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:46329 every 8 connection(s)
I20260924 07:18:43.683787 8747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:43.686946 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.684581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:49527 (local address 127.3.246.126:42691)
0924 07:18:43.684644 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.684646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.684922 (+ 276us) server_negotiation.cc:407] Connection header received
0924 07:18:43.685450 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.685452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.685459 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.685510 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:43.686678 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:43.686680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:43.686680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:43.686685 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:43.686700 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:43.686705 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:43.686760 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.686863 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.686878 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:18:43.687140 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.684574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42691 (local address 127.3.246.65:49527)
0924 07:18:43.684778 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.684809 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.685323 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.685579 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.685587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.685595 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:43.686531 (+ 936us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:18:43.686531 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:43.686558 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:18:43.686585 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:43.686768 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:43.686773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:43.686775 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.686874 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.686932 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260924 07:18:43.687668 8875 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42691
I20260924 07:18:43.687703 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 8747
I20260924 07:18:43.687767 8875 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:43.687794 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:43.687927 8875 heartbeater.cc:511] Master 127.3.246.126:42691 requested a full tablet report, sending...
I20260924 07:18:43.687988 4057 external_mini_cluster.cc:1428] Setting key d4b1dd798ba454d061543c7752c6d3f4
I20260924 07:18:43.688493 8687 ts_manager.cc:194] Registered new tserver with Master: 2a0788878b7f4d49b95817eeec81f863 (127.3.246.65:46329)
I20260924 07:18:43.689177 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42691
--builtin_ntp_servers=127.3.246.84:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:18:43.770231 8878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:43.770449 8878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:43.770473 8878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:43.771596 8878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:43.771631 8878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:43.771656 8878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:43.772894 8878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:43.773069 8878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:43.773307 8878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:43.778640 8884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:43.778698 8886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:43.778961 8878 server_base.cc:1034] running on GCE node
W20260924 07:18:43.778647 8883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:43.780004 8878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:43.780211 8878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:43.781351 8878 hybrid_clock.cc:648] HybridClock initialized: now 1790234323781342 us; error 31 us; skew 500 ppm
I20260924 07:18:43.782758 8878 webserver.cc:460] Webserver started at http://127.3.246.66:39445/ using document root <none> and password file <none>
I20260924 07:18:43.783004 8878 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:43.783053 8878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:43.783226 8878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:43.784293 8878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "0578f2180bc74a5c999a4f3c4b16fd8c"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "70d0740271d4015b31e1e0870af64d3b"
server_key_iv: "78b1c2f491ba65f3012ad4fe03c41c30"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.784670 8878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "0578f2180bc74a5c999a4f3c4b16fd8c"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "70d0740271d4015b31e1e0870af64d3b"
server_key_iv: "78b1c2f491ba65f3012ad4fe03c41c30"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.786008 8878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:43.786787 8892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:43.786996 8878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:43.787102 8878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "0578f2180bc74a5c999a4f3c4b16fd8c"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "70d0740271d4015b31e1e0870af64d3b"
server_key_iv: "78b1c2f491ba65f3012ad4fe03c41c30"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.787191 8878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:43.790562 8878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:43.790829 8878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:43.790957 8878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:43.791168 8878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:43.791481 8878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:43.791513 8878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:43.791567 8878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:43.791594 8878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:43.797304 8878 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:45169
I20260924 07:18:43.797346 9005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:45169 every 8 connection(s)
I20260924 07:18:43.797713 8878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:43.800675 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.798515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:44781 (local address 127.3.246.126:42691)
0924 07:18:43.798577 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.798580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.798780 (+ 200us) server_negotiation.cc:407] Connection header received
0924 07:18:43.799218 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.799219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.799224 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.799300 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:43.800489 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:43.800490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:43.800490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:43.800494 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:43.800509 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:43.800513 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:43.800563 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.800608 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.800620 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:18:43.800868 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.798471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42691 (local address 127.3.246.66:44781)
0924 07:18:43.798649 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.798688 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.799130 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.799353 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.799356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.799370 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:43.800329 (+ 959us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:18:43.800329 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:43.800357 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:18:43.800394 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:43.800560 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:43.800562 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:43.800564 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.800614 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.800659 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260924 07:18:43.801432 9006 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42691
I20260924 07:18:43.801510 9006 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:43.801614 9006 heartbeater.cc:511] Master 127.3.246.126:42691 requested a full tablet report, sending...
I20260924 07:18:43.802006 8687 ts_manager.cc:194] Registered new tserver with Master: 0578f2180bc74a5c999a4f3c4b16fd8c (127.3.246.66:45169)
I20260924 07:18:43.803630 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 8878
I20260924 07:18:43.803733 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:43.803913 4057 external_mini_cluster.cc:1428] Setting key 5afa5e285bfe2b711bcbcaad20dc6711
I20260924 07:18:43.804818 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42691
--builtin_ntp_servers=127.3.246.84:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:18:43.884279 9009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:43.884478 9009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:43.884502 9009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:43.885689 9009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:43.885730 9009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:43.885762 9009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:43.887187 9009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:43.887425 9009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:43.887712 9009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:43.892894 9015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:43.892951 9014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:43.892923 9017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:43.893442 9009 server_base.cc:1034] running on GCE node
I20260924 07:18:43.893610 9009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:43.893851 9009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:43.895036 9009 hybrid_clock.cc:648] HybridClock initialized: now 1790234323895010 us; error 34 us; skew 500 ppm
I20260924 07:18:43.896306 9009 webserver.cc:460] Webserver started at http://127.3.246.67:43893/ using document root <none> and password file <none>
I20260924 07:18:43.896543 9009 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:43.896613 9009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:43.896777 9009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:43.897921 9009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "a718851d43d94f89acc913b00c2668b3"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "2d8288eee29621ea4ef037a25276cb91"
server_key_iv: "239aec84b0bc7defcb7dcb138fa0c9a9"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.898293 9009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "a718851d43d94f89acc913b00c2668b3"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "2d8288eee29621ea4ef037a25276cb91"
server_key_iv: "239aec84b0bc7defcb7dcb138fa0c9a9"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.899590 9009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:43.900293 9023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:43.900462 9009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:43.900558 9009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "a718851d43d94f89acc913b00c2668b3"
format_stamp: "Formatted at 2026-09-24 07:18:43 on dist-test-slave-rc1w"
server_key: "2d8288eee29621ea4ef037a25276cb91"
server_key_iv: "239aec84b0bc7defcb7dcb138fa0c9a9"
server_key_version: "encryptionkey@0"
I20260924 07:18:43.900640 9009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:43.904016 9009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:43.904330 9009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:43.904470 9009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:43.904676 9009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:43.904968 9009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:43.905021 9009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:43.905058 9009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:43.905086 9009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:43.910703 9009 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:32931
I20260924 07:18:43.910772 9136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:32931 every 8 connection(s)
I20260924 07:18:43.911123 9009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:43.914419 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.911876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:39729 (local address 127.3.246.126:42691)
0924 07:18:43.911968 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.911971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.912159 (+ 188us) server_negotiation.cc:407] Connection header received
0924 07:18:43.912726 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.912728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.912735 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.912799 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:43.914144 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:43.914146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:43.914146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:43.914151 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:43.914169 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:43.914173 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:43.914233 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.914334 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.914350 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:18:43.914628 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.911828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42691 (local address 127.3.246.67:39729)
0924 07:18:43.912026 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.912070 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.912548 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.912870 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.912874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.912883 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:43.913994 (+ 1111us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:18:43.913995 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:43.914023 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:18:43.914050 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:43.914233 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:43.914238 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:43.914241 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.914330 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.914391 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260924 07:18:43.915287 9137 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42691
I20260924 07:18:43.915395 9137 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:43.915555 9137 heartbeater.cc:511] Master 127.3.246.126:42691 requested a full tablet report, sending...
I20260924 07:18:43.915925 8687 ts_manager.cc:194] Registered new tserver with Master: a718851d43d94f89acc913b00c2668b3 (127.3.246.67:32931)
I20260924 07:18:43.919353 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 9009
I20260924 07:18:43.919468 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:43.919653 4057 external_mini_cluster.cc:1428] Setting key 07a8a2c4c8bc0bc064da1d88785ce1bb
I20260924 07:18:43.920491 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:18:43.922685 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.921734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.3.246.126:42691)
0924 07:18:43.921793 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.921794 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.921935 (+ 141us) server_negotiation.cc:407] Connection header received
0924 07:18:43.921975 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.921975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.921980 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.922032 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:43.922531 (+ 499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:43.922532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:43.922532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:43.922535 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:43.922548 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:43.922552 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:43.922590 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.922627 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.922637 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:18:43.924382 8687 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:43.926034 8687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:43.929833 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.928174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:46329 (local address 127.0.0.1:37978)
0924 07:18:43.928417 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.928427 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.928464 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.928888 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.928889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.928892 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:43.929357 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:18:43.929358 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:43.929374 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:18:43.929398 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:43.929639 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:43.929645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:43.929648 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.929741 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.929755 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":76,"threads_started":1}
I20260924 07:18:43.930017 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.928231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.3.246.66:45169)
0924 07:18:43.928557 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.928562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.928573 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:18:43.928599 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.928602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.928611 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.928695 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:43.929638 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:43.929648 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:43.929648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:43.929653 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:43.929722 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:43.929752 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:43.929820 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.929891 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.929909 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":86,"threads_started":1}
I20260924 07:18:43.930127 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.928361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:32931 (local address 127.0.0.1:44218)
0924 07:18:43.928545 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.928554 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.928584 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.929115 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.929116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.929118 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:43.929754 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:18:43.929755 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:43.929762 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:18:43.929773 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:43.929997 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:43.929997 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:43.929998 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.930042 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.930054 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":64,"threads_started":1}
I20260924 07:18:43.930141 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.928291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.3.246.65:46329)
0924 07:18:43.928730 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.928735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.928745 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:18:43.928772 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.928775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.928785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.929052 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:43.929476 (+ 424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:43.929485 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:43.929485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:43.929489 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:43.929523 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:43.929542 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:43.929641 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.930018 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.930034 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
I20260924 07:18:43.930423 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.928139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:45169 (local address 127.0.0.1:37790)
0924 07:18:43.928331 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.928373 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.928432 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.928697 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.928702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.928707 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:43.929522 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:18:43.929522 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:43.929529 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:18:43.929541 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:43.929814 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:43.929814 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:43.929815 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.929870 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.930353 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
I20260924 07:18:43.930519 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.928539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.3.246.67:32931)
0924 07:18:43.928700 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.928705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.928717 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:18:43.928744 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.928746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.928756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.928837 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:43.929856 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:43.929866 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:43.929866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:43.929870 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:43.929901 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:43.929922 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:43.929973 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.930412 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.930426 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260924 07:18:43.930840 8809 tablet_service.cc:1467] Processing CreateTablet for tablet 71e750fc156a4d2cbb83cdf587718cd9 (DEFAULT_TABLE table=test-table [id=40e841ac9a2d42c1a5ba91d16a92cb9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:43.931042 8940 tablet_service.cc:1467] Processing CreateTablet for tablet 71e750fc156a4d2cbb83cdf587718cd9 (DEFAULT_TABLE table=test-table [id=40e841ac9a2d42c1a5ba91d16a92cb9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:43.931211 8809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e750fc156a4d2cbb83cdf587718cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:43.931088 9071 tablet_service.cc:1467] Processing CreateTablet for tablet 71e750fc156a4d2cbb83cdf587718cd9 (DEFAULT_TABLE table=test-table [id=40e841ac9a2d42c1a5ba91d16a92cb9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:43.931339 8940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e750fc156a4d2cbb83cdf587718cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:43.931456 9071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e750fc156a4d2cbb83cdf587718cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:43.934651 9152 tablet_bootstrap.cc:492] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c: Bootstrap starting.
I20260924 07:18:43.934755 9151 tablet_bootstrap.cc:492] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3: Bootstrap starting.
I20260924 07:18:43.934957 9153 tablet_bootstrap.cc:492] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863: Bootstrap starting.
I20260924 07:18:43.935309 9152 tablet_bootstrap.cc:654] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:43.935490 9151 tablet_bootstrap.cc:654] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:43.935564 9153 tablet_bootstrap.cc:654] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:43.935659 9152 log.cc:824] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:43.935868 9153 log.cc:824] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:43.935875 9151 log.cc:824] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:43.936614 9153 tablet_bootstrap.cc:492] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863: No bootstrap required, opened a new log
I20260924 07:18:43.936614 9152 tablet_bootstrap.cc:492] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c: No bootstrap required, opened a new log
I20260924 07:18:43.936684 9153 ts_tablet_manager.cc:1397] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:43.936697 9152 ts_tablet_manager.cc:1397] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:43.937999 9151 tablet_bootstrap.cc:492] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3: No bootstrap required, opened a new log
I20260924 07:18:43.938082 9151 ts_tablet_manager.cc:1397] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 07:18:43.939307 9152 raft_consensus.cc:348] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0788878b7f4d49b95817eeec81f863" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46329 } } peers { permanent_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 45169 } } peers { permanent_uuid: "a718851d43d94f89acc913b00c2668b3" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 32931 } }
I20260924 07:18:43.939308 9153 raft_consensus.cc:348] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0788878b7f4d49b95817eeec81f863" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46329 } } peers { permanent_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 45169 } } peers { permanent_uuid: "a718851d43d94f89acc913b00c2668b3" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 32931 } }
I20260924 07:18:43.939474 9153 raft_consensus.cc:374] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:43.939474 9152 raft_consensus.cc:374] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:43.939512 9153 raft_consensus.cc:729] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0788878b7f4d49b95817eeec81f863, State: Initialized, Role: FOLLOWER
I20260924 07:18:43.939512 9152 raft_consensus.cc:729] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0578f2180bc74a5c999a4f3c4b16fd8c, State: Initialized, Role: FOLLOWER
I20260924 07:18:43.939626 9153 consensus_queue.cc:260] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0788878b7f4d49b95817eeec81f863" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46329 } } peers { permanent_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 45169 } } peers { permanent_uuid: "a718851d43d94f89acc913b00c2668b3" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 32931 } }
I20260924 07:18:43.939640 9152 consensus_queue.cc:260] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0788878b7f4d49b95817eeec81f863" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46329 } } peers { permanent_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 45169 } } peers { permanent_uuid: "a718851d43d94f89acc913b00c2668b3" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 32931 } }
I20260924 07:18:43.939890 9153 ts_tablet_manager.cc:1428] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 07:18:43.939890 9152 ts_tablet_manager.cc:1428] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:43.940004 8875 heartbeater.cc:503] Master 127.3.246.126:42691 was elected leader, sending a full tablet report...
I20260924 07:18:43.940070 9006 heartbeater.cc:503] Master 127.3.246.126:42691 was elected leader, sending a full tablet report...
I20260924 07:18:43.940312 9151 raft_consensus.cc:348] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0788878b7f4d49b95817eeec81f863" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46329 } } peers { permanent_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 45169 } } peers { permanent_uuid: "a718851d43d94f89acc913b00c2668b3" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 32931 } }
I20260924 07:18:43.940471 9151 raft_consensus.cc:374] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:43.940505 9151 raft_consensus.cc:729] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a718851d43d94f89acc913b00c2668b3, State: Initialized, Role: FOLLOWER
I20260924 07:18:43.940613 9151 consensus_queue.cc:260] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0788878b7f4d49b95817eeec81f863" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46329 } } peers { permanent_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 45169 } } peers { permanent_uuid: "a718851d43d94f89acc913b00c2668b3" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 32931 } }
I20260924 07:18:43.940886 9151 ts_tablet_manager.cc:1428] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:43.940996 9137 heartbeater.cc:503] Master 127.3.246.126:42691 was elected leader, sending a full tablet report...
I20260924 07:18:44.041179 9157 raft_consensus.cc:482] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:44.041317 9157 raft_consensus.cc:504] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0788878b7f4d49b95817eeec81f863" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46329 } } peers { permanent_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 45169 } } peers { permanent_uuid: "a718851d43d94f89acc913b00c2668b3" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 32931 } }
I20260924 07:18:44.041781 9157 leader_election.cc:288] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a0788878b7f4d49b95817eeec81f863 (127.3.246.65:46329), a718851d43d94f89acc913b00c2668b3 (127.3.246.67:32931)
I20260924 07:18:44.043206 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.042114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:39101 (local address 127.3.246.65:46329)
0924 07:18:44.042161 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:44.042163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:44.042174 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:18:44.042226 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:44.042227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:44.042232 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:44.042301 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:44.042929 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:44.042931 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:44.042931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:44.042936 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:44.042957 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:44.042962 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:44.043027 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:44.043119 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 07:18:44.043135 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:18:44.043200 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.042020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:46329 (local address 127.3.246.66:39101)
0924 07:18:44.042084 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:44.042095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:44.042137 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:44.042323 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:44.042324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:44.042326 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:44.042815 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:18:44.042816 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:44.042824 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:18:44.042838 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:44.043038 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:44.043039 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:44.043040 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 07:18:44.043112 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:18:44.043128 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:18:44.043298 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.042127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:58729 (local address 127.3.246.67:32931)
0924 07:18:44.042187 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:44.042190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:44.042311 (+ 121us) server_negotiation.cc:407] Connection header received
0924 07:18:44.042388 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:44.042389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:44.042395 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:44.042453 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:44.043084 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:44.043086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:44.043086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:44.043090 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:44.043103 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:44.043107 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:44.043156 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:44.043206 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 07:18:44.043220 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:18:44.043309 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.042055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:32931 (local address 127.3.246.66:58729)
0924 07:18:44.042234 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:44.042248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:44.042321 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:44.042457 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:44.042458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:44.042471 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:44.043005 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:18:44.043006 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:44.043012 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:18:44.043024 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:44.043152 (+ 128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:44.043152 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:44.043153 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 07:18:44.043206 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:18:44.043220 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
I20260924 07:18:44.043555 8829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e750fc156a4d2cbb83cdf587718cd9" candidate_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a0788878b7f4d49b95817eeec81f863" is_pre_election: true
I20260924 07:18:44.043674 8829 raft_consensus.cc:2454] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0578f2180bc74a5c999a4f3c4b16fd8c in term 0.
I20260924 07:18:44.043927 8895 leader_election.cc:302] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0578f2180bc74a5c999a4f3c4b16fd8c, 2a0788878b7f4d49b95817eeec81f863; no voters:
I20260924 07:18:44.043965 9091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e750fc156a4d2cbb83cdf587718cd9" candidate_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a718851d43d94f89acc913b00c2668b3" is_pre_election: true
I20260924 07:18:44.044095 9091 raft_consensus.cc:2454] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0578f2180bc74a5c999a4f3c4b16fd8c in term 0.
I20260924 07:18:44.044109 9157 raft_consensus.cc:2781] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:44.044186 9157 raft_consensus.cc:482] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:44.044214 9157 raft_consensus.cc:3037] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:44.044953 9157 raft_consensus.cc:504] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0788878b7f4d49b95817eeec81f863" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46329 } } peers { permanent_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 45169 } } peers { permanent_uuid: "a718851d43d94f89acc913b00c2668b3" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 32931 } }
I20260924 07:18:44.045104 9157 leader_election.cc:288] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [CANDIDATE]: Term 1 election: Requested vote from peers 2a0788878b7f4d49b95817eeec81f863 (127.3.246.65:46329), a718851d43d94f89acc913b00c2668b3 (127.3.246.67:32931)
I20260924 07:18:44.045295 9091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e750fc156a4d2cbb83cdf587718cd9" candidate_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a718851d43d94f89acc913b00c2668b3"
I20260924 07:18:44.045352 8829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e750fc156a4d2cbb83cdf587718cd9" candidate_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a0788878b7f4d49b95817eeec81f863"
I20260924 07:18:44.045382 9091 raft_consensus.cc:3037] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:44.045461 8829 raft_consensus.cc:3037] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:44.046241 9091 raft_consensus.cc:2454] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0578f2180bc74a5c999a4f3c4b16fd8c in term 1.
I20260924 07:18:44.046259 8829 raft_consensus.cc:2454] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0578f2180bc74a5c999a4f3c4b16fd8c in term 1.
I20260924 07:18:44.046423 8893 leader_election.cc:302] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0578f2180bc74a5c999a4f3c4b16fd8c, a718851d43d94f89acc913b00c2668b3; no voters:
I20260924 07:18:44.046555 9157 raft_consensus.cc:2781] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:44.046708 9157 raft_consensus.cc:686] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [term 1 LEADER]: Becoming Leader. State: Replica: 0578f2180bc74a5c999a4f3c4b16fd8c, State: Running, Role: LEADER
I20260924 07:18:44.046824 9157 consensus_queue.cc:237] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0788878b7f4d49b95817eeec81f863" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46329 } } peers { permanent_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 45169 } } peers { permanent_uuid: "a718851d43d94f89acc913b00c2668b3" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 32931 } }
I20260924 07:18:44.047621 8687 catalog_manager.cc:5495] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0578f2180bc74a5c999a4f3c4b16fd8c (127.3.246.66). New cstate: current_term: 1 leader_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0788878b7f4d49b95817eeec81f863" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0578f2180bc74a5c999a4f3c4b16fd8c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 45169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a718851d43d94f89acc913b00c2668b3" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
W20260924 07:18:44.048249 9007 tablet.cc:2369] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:18:44.078935 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.077905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.3.246.66:45169)
0924 07:18:44.077968 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:44.077970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:44.077986 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:18:44.078023 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:44.078023 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:44.078036 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:44.078093 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:44.078714 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:44.078716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:44.078717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:44.078722 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:44.078740 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:44.078744 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:44.078800 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:44.078866 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 07:18:44.078878 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:18:44.080726 8829 raft_consensus.cc:1261] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863 [term 1 FOLLOWER]: Refusing update from remote peer 0578f2180bc74a5c999a4f3c4b16fd8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:44.080756 9091 raft_consensus.cc:1261] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3 [term 1 FOLLOWER]: Refusing update from remote peer 0578f2180bc74a5c999a4f3c4b16fd8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:44.081102 9161 consensus_queue.cc:1035] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a0788878b7f4d49b95817eeec81f863" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:44.081235 9157 consensus_queue.cc:1035] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a718851d43d94f89acc913b00c2668b3" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:44.084205 9165 mvcc.cc:205] Tried to move back new op lower bound from 7332799791432085504 to 7332799791296458752. Current Snapshot: MvccSnapshot[applied={T|T < 7332799791432085504}]
I20260924 07:18:44.087085 8687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:44.087194 8687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:44.088018 8687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d2ff93b07e84a82bf0910c37ca1dddf: Sending DeleteTablet for 3 replicas of tablet 71e750fc156a4d2cbb83cdf587718cd9
I20260924 07:18:44.088352 8809 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e750fc156a4d2cbb83cdf587718cd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:44 UTC) from {username='slave'} at 127.0.0.1:37978
I20260924 07:18:44.088366 9071 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e750fc156a4d2cbb83cdf587718cd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:44 UTC) from {username='slave'} at 127.0.0.1:44218
I20260924 07:18:44.088573 8940 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e750fc156a4d2cbb83cdf587718cd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:44 UTC) from {username='slave'} at 127.0.0.1:37790
I20260924 07:18:44.088814 9176 tablet_replica.cc:330] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c: stopping tablet replica
I20260924 07:18:44.088872 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 8747
I20260924 07:18:44.088941 9174 tablet_replica.cc:330] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3: stopping tablet replica
I20260924 07:18:44.089042 9176 raft_consensus.cc:2229] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:44.089174 9175 tablet_replica.cc:330] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863: stopping tablet replica
I20260924 07:18:44.089274 9175 raft_consensus.cc:2229] T 71e750fc156a4d2cbb83cdf587718cd9 P 2a0788878b7f4d49b95817eeec81f863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:44.089455 9174 raft_consensus.cc:2229] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:44.089573 9174 raft_consensus.cc:2258] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:44.089866 9176 raft_consensus.cc:2258] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:44.090381 9176 ts_tablet_manager.cc:1905] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:44.092247 9176 ts_tablet_manager.cc:1918] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:44.092324 9176 log.cc:1196] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/71e750fc156a4d2cbb83cdf587718cd9
I20260924 07:18:44.092613 9176 ts_tablet_manager.cc:1939] T 71e750fc156a4d2cbb83cdf587718cd9 P 0578f2180bc74a5c999a4f3c4b16fd8c: Deleting consensus metadata
I20260924 07:18:44.093019 8671 catalog_manager.cc:4841] TS 0578f2180bc74a5c999a4f3c4b16fd8c (127.3.246.66:45169): tablet 71e750fc156a4d2cbb83cdf587718cd9 (table test-table [id=40e841ac9a2d42c1a5ba91d16a92cb9b]) successfully deleted
W20260924 07:18:44.097826 8673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:46329 (error 108)
I20260924 07:18:44.098066 9174 ts_tablet_manager.cc:1905] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:44.098323 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.098198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:46329 (local address 127.0.0.1:37986)
0924 07:18:44.098248 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:44.098288 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:46329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 07:18:44.098315 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 8878
W20260924 07:18:44.098423 8673 catalog_manager.cc:4584] TS 2a0788878b7f4d49b95817eeec81f863 (127.3.246.65:46329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71e750fc156a4d2cbb83cdf587718cd9: Network error: Client connection negotiation failed: client connection to 127.3.246.65:46329: connect: Connection refused (error 111)
I20260924 07:18:44.099920 9174 ts_tablet_manager.cc:1918] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:44.099998 9174 log.cc:1196] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/71e750fc156a4d2cbb83cdf587718cd9
I20260924 07:18:44.100272 9174 ts_tablet_manager.cc:1939] T 71e750fc156a4d2cbb83cdf587718cd9 P a718851d43d94f89acc913b00c2668b3: Deleting consensus metadata
I20260924 07:18:44.100735 8671 catalog_manager.cc:4841] TS a718851d43d94f89acc913b00c2668b3 (127.3.246.67:32931): tablet 71e750fc156a4d2cbb83cdf587718cd9 (table test-table [id=40e841ac9a2d42c1a5ba91d16a92cb9b]) successfully deleted
I20260924 07:18:44.103271 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 9009
I20260924 07:18:44.107752 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 8656
2026-09-24T07:18:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (712 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 07:18:44.127261 4057 mini_oidc.cc:152] Starting JWKS server
I20260924 07:18:44.128335 4057 webserver.cc:460] Webserver started at https://127.0.0.1:41829/ using document root <none> and password file <none>
I20260924 07:18:44.128456 4057 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:18:44.128638 4057 webserver.cc:460] Webserver started at http://127.0.0.1:34893/ using document root <none> and password file <none>
2026-09-24T07:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:44Z Disabled control of system clock
I20260924 07:18:44.136144 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:39235
--webserver_interface=127.3.246.126
--webserver_port=0
--builtin_ntp_servers=127.3.246.84:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:39235
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41829/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/testchainca.cert with env {}
W20260924 07:18:44.214964 9183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:44.215165 9183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:44.215188 9183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:44.216298 9183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:44.216336 9183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:44.216348 9183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:44.216360 9183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:44.216372 9183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:18:44.216384 9183 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:18:44.216396 9183 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41829/jwks.json
I20260924 07:18:44.217950 9183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:39235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:39235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41829/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:44.218137 9183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:44.218298 9183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:44.223107 9188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:44.223199 9183 server_base.cc:1034] running on GCE node
W20260924 07:18:44.223170 9189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:44.223178 9191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:44.223620 9183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:44.223865 9183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:44.225003 9183 hybrid_clock.cc:648] HybridClock initialized: now 1790234324224993 us; error 35 us; skew 500 ppm
I20260924 07:18:44.226351 9183 webserver.cc:460] Webserver started at http://127.3.246.126:43591/ using document root <none> and password file <none>
I20260924 07:18:44.226539 9183 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:44.226579 9183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:44.226667 9183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:44.227665 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "8a0f7e81b2bf4c49b9789885863ec3e6"
format_stamp: "Formatted at 2026-09-24 07:18:44 on dist-test-slave-rc1w"
server_key: "62e21f966286e1184d4ab9d869146b24"
server_key_iv: "7b113d18aaf7d402d2da077593b580fb"
server_key_version: "encryptionkey@0"
I20260924 07:18:44.227957 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "8a0f7e81b2bf4c49b9789885863ec3e6"
format_stamp: "Formatted at 2026-09-24 07:18:44 on dist-test-slave-rc1w"
server_key: "62e21f966286e1184d4ab9d869146b24"
server_key_iv: "7b113d18aaf7d402d2da077593b580fb"
server_key_version: "encryptionkey@0"
I20260924 07:18:44.229162 9183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:18:44.230001 9197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:44.230193 9183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:44.230288 9183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "8a0f7e81b2bf4c49b9789885863ec3e6"
format_stamp: "Formatted at 2026-09-24 07:18:44 on dist-test-slave-rc1w"
server_key: "62e21f966286e1184d4ab9d869146b24"
server_key_iv: "7b113d18aaf7d402d2da077593b580fb"
server_key_version: "encryptionkey@0"
I20260924 07:18:44.230343 9183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:44.240149 9183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:44.240406 9183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:44.240504 9183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:44.244280 9183 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:39235
I20260924 07:18:44.244315 9249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:39235 every 8 connection(s)
I20260924 07:18:44.244668 9183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:44.245632 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:44.247972 9250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6: Bootstrap starting.
I20260924 07:18:44.248580 9250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:44.248893 9250 log.cc:824] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:44.249619 9250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6: No bootstrap required, opened a new log
I20260924 07:18:44.250346 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 9183
I20260924 07:18:44.250437 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:44.250689 4057 external_mini_cluster.cc:1428] Setting key 48c835bc48accb32676093f2433e410e
I20260924 07:18:44.251866 9250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0f7e81b2bf4c49b9789885863ec3e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 39235 } }
I20260924 07:18:44.252039 9250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:44.252075 9250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a0f7e81b2bf4c49b9789885863ec3e6, State: Initialized, Role: FOLLOWER
I20260924 07:18:44.252225 9250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0f7e81b2bf4c49b9789885863ec3e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 39235 } }
I20260924 07:18:44.252307 9250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:44.252346 9250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:44.252396 9250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:44.253109 9250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0f7e81b2bf4c49b9789885863ec3e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 39235 } }
I20260924 07:18:44.253216 9250 leader_election.cc:302] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a0f7e81b2bf4c49b9789885863ec3e6; no voters:
I20260924 07:18:44.253377 9250 leader_election.cc:288] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:44.253448 9255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:44.253603 9250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:44.253615 9255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [term 1 LEADER]: Becoming Leader. State: Replica: 8a0f7e81b2bf4c49b9789885863ec3e6, State: Running, Role: LEADER
I20260924 07:18:44.253755 9255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0f7e81b2bf4c49b9789885863ec3e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 39235 } }
I20260924 07:18:44.254082 9255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a0f7e81b2bf4c49b9789885863ec3e6. Latest consensus state: current_term: 1 leader_uuid: "8a0f7e81b2bf4c49b9789885863ec3e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0f7e81b2bf4c49b9789885863ec3e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 39235 } } }
I20260924 07:18:44.254172 9255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:44.254091 9257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a0f7e81b2bf4c49b9789885863ec3e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0f7e81b2bf4c49b9789885863ec3e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 39235 } } }
I20260924 07:18:44.254328 9257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:44.254488 9263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:44.255012 9263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:44.257431 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.251227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.3.246.126:39235)
0924 07:18:44.251453 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:44.251463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:44.251495 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:18:44.251824 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:44.251840 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:44.251918 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:44.252066 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:44.252313 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.254122 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.255336 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.255937 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.256053 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:44.256736 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:44.256739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:44.256740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:44.256951 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:44.257000 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:44.257014 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:44.257085 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:44.257144 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 07:18:44.257223 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260924 07:18:44.258455 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:18:44.259167 9263 catalog_manager.cc:1347] Generated new cluster ID: db3b38e440a6405e85940b769471a796
I20260924 07:18:44.259215 9263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:44.269156 9263 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:44.269750 9263 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:44.277105 9263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a0f7e81b2bf4c49b9789885863ec3e6: Generated new TSK 0
I20260924 07:18:44.277299 9263 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 07:18:44.302897 9280 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 07:18:44.303012 9280 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260924 07:18:44.303598 9279 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:44.287307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:39235 (local address 127.0.0.1:51308)
0924 07:18:44.287512 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:44.287523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:44.287586 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:44.287784 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:44.287785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:44.287787 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:18:44.287863 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:44.287866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.288833 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.288834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:44.289434 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:44.289437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.289952 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.289953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:44.290012 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:44.290031 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:18:44.303163 (+ 13132us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 07:18:44.303263 (+ 100us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.246.126:39235: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260924 07:18:44.303908 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.287440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.3.246.126:39235)
0924 07:18:44.287506 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:44.287509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:44.287583 (+ 74us) server_negotiation.cc:407] Connection header received
0924 07:18:44.287668 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:44.287669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:44.287680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:44.287761 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:18:44.287936 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.288733 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.289523 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.289854 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.289923 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:44.290138 (+ 215us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:18:44.303015 (+ 12877us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 07:18:44.303843 (+ 828us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51308: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 07:18:44.303954 9254 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51308: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.246.126:39235: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 07:18:44.304224 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 9183
2026-09-24T07:18:44Z chronyd exiting
I20260924 07:18:44.340241 4057 test_util.cc:175] -----------------------------------------------
I20260924 07:18:44.340343 4057 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234301618289-4057-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 07:18:44.341097 4057 mini_oidc.cc:152] Starting JWKS server
I20260924 07:18:44.341858 4057 webserver.cc:460] Webserver started at https://127.0.0.1:39865/ using document root <none> and password file <none>
I20260924 07:18:44.341926 4057 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:18:44.342083 4057 webserver.cc:460] Webserver started at http://127.0.0.1:45571/ using document root <none> and password file <none>
2026-09-24T07:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:44Z Disabled control of system clock
I20260924 07:18:44.349964 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:44973
--webserver_interface=127.3.246.126
--webserver_port=0
--builtin_ntp_servers=127.3.246.84:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:44973
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39865/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/testchainca.cert with env {}
W20260924 07:18:44.430683 9286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:44.430903 9286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:44.430927 9286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:44.432031 9286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:44.432071 9286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:44.432086 9286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:44.432101 9286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:44.432114 9286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:18:44.432140 9286 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:18:44.432154 9286 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39865/jwks.json
I20260924 07:18:44.433729 9286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:44973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:44973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39865/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:44.433926 9286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:44.434093 9286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:44.439009 9292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:44.439112 9286 server_base.cc:1034] running on GCE node
W20260924 07:18:44.439009 9291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:44.439037 9294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:44.439519 9286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:44.439785 9286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:44.440964 9286 hybrid_clock.cc:648] HybridClock initialized: now 1790234324440951 us; error 26 us; skew 500 ppm
I20260924 07:18:44.442235 9286 webserver.cc:460] Webserver started at http://127.3.246.126:46013/ using document root <none> and password file <none>
I20260924 07:18:44.442466 9286 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:44.442535 9286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:44.442709 9286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:44.443782 9286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "0ff99d4c4fd84368a0345a1c5d721197"
format_stamp: "Formatted at 2026-09-24 07:18:44 on dist-test-slave-rc1w"
server_key: "c6b18817bb3ffa32a2c472048704c5c9"
server_key_iv: "b790db551f754a6a6ad7a69e5893a347"
server_key_version: "encryptionkey@0"
I20260924 07:18:44.444133 9286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "0ff99d4c4fd84368a0345a1c5d721197"
format_stamp: "Formatted at 2026-09-24 07:18:44 on dist-test-slave-rc1w"
server_key: "c6b18817bb3ffa32a2c472048704c5c9"
server_key_iv: "b790db551f754a6a6ad7a69e5893a347"
server_key_version: "encryptionkey@0"
I20260924 07:18:44.445386 9286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:18:44.446190 9300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:44.446359 9286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:44.446456 9286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "0ff99d4c4fd84368a0345a1c5d721197"
format_stamp: "Formatted at 2026-09-24 07:18:44 on dist-test-slave-rc1w"
server_key: "c6b18817bb3ffa32a2c472048704c5c9"
server_key_iv: "b790db551f754a6a6ad7a69e5893a347"
server_key_version: "encryptionkey@0"
I20260924 07:18:44.446540 9286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:44.467231 9286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:44.467571 9286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:44.467742 9286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:44.471930 9286 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:44973
I20260924 07:18:44.471987 9352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:44973 every 8 connection(s)
I20260924 07:18:44.472339 9286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:44.473276 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:44.474126 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 9286
I20260924 07:18:44.474242 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:44.474436 4057 external_mini_cluster.cc:1428] Setting key ec9ba23d9115d01888ee582ead2eefe3
I20260924 07:18:44.476686 9353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197: Bootstrap starting.
I20260924 07:18:44.477385 9353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:44.477735 9353 log.cc:824] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:44.478461 9353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197: No bootstrap required, opened a new log
I20260924 07:18:44.480643 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.474969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.3.246.126:44973)
0924 07:18:44.475193 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:44.475203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:44.475236 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:18:44.475816 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:44.475826 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:44.475910 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:44.476062 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:44.476233 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.477556 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.478437 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.479021 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.479156 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:44.479884 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:44.479889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:44.479889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:44.480109 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:44.480172 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:44.480190 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:44.480271 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:44.480335 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 07:18:44.480434 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260924 07:18:44.480741 9353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff99d4c4fd84368a0345a1c5d721197" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44973 } }
I20260924 07:18:44.480899 9353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:44.480933 9353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff99d4c4fd84368a0345a1c5d721197, State: Initialized, Role: FOLLOWER
I20260924 07:18:44.481077 9353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff99d4c4fd84368a0345a1c5d721197" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44973 } }
I20260924 07:18:44.481169 9353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:44.481225 9353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:44.481272 9353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:44.481693 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:18:44.481925 9353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff99d4c4fd84368a0345a1c5d721197" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44973 } }
I20260924 07:18:44.482023 9353 leader_election.cc:302] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ff99d4c4fd84368a0345a1c5d721197; no voters:
I20260924 07:18:44.482172 9353 leader_election.cc:288] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:44.482239 9358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:44.482414 9353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:44.482420 9358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [term 1 LEADER]: Becoming Leader. State: Replica: 0ff99d4c4fd84368a0345a1c5d721197, State: Running, Role: LEADER
I20260924 07:18:44.482561 9358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff99d4c4fd84368a0345a1c5d721197" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44973 } }
I20260924 07:18:44.483074 9358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ff99d4c4fd84368a0345a1c5d721197. Latest consensus state: current_term: 1 leader_uuid: "0ff99d4c4fd84368a0345a1c5d721197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff99d4c4fd84368a0345a1c5d721197" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44973 } } }
I20260924 07:18:44.483091 9359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ff99d4c4fd84368a0345a1c5d721197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff99d4c4fd84368a0345a1c5d721197" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44973 } } }
I20260924 07:18:44.483165 9358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:44.483228 9359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:44.483436 9367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:44.483944 9367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:44.485491 9367 catalog_manager.cc:1347] Generated new cluster ID: 4073a7f78a174546b9bc05b9990fabf1
I20260924 07:18:44.485551 9367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:44.510601 9367 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:44.511330 9367 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:44.516714 9367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ff99d4c4fd84368a0345a1c5d721197: Generated new TSK 0
I20260924 07:18:44.516902 9367 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 07:18:44.544507 9383 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 07:18:44.544603 9383 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 07:18:44.544917 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.523391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.3.246.126:44973)
0924 07:18:44.523435 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:44.523437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:44.523538 (+ 101us) server_negotiation.cc:407] Connection header received
0924 07:18:44.523625 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:44.523626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:44.523637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:44.523698 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:18:44.523866 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.524380 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.525498 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.525745 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.525804 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:44.526040 (+ 236us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:18:44.544676 (+ 18636us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 07:18:44.544860 (+ 184us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 07:18:44.544996 9356 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 07:18:44.545073 9382 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:44.523292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:44973 (local address 127.0.0.1:38242)
0924 07:18:44.523457 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:44.523470 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:44.523546 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:44.523719 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:44.523720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:44.523722 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:18:44.523789 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:44.523792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.524461 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.524462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:44.525411 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:44.525415 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.525826 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.525827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:44.525894 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:44.525900 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:18:44.544833 (+ 18933us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 07:18:44.544959 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.246.126:44973: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260924 07:18:44.545370 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 9286
2026-09-24T07:18:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (242 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 07:18:44.583195 4057 mini_oidc.cc:152] Starting JWKS server
I20260924 07:18:44.583977 4057 webserver.cc:460] Webserver started at https://127.0.0.1:42803/ using document root <none> and password file <none>
I20260924 07:18:44.584036 4057 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:18:44.584255 4057 webserver.cc:460] Webserver started at http://127.0.0.1:44687/ using document root <none> and password file <none>
2026-09-24T07:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:44Z Disabled control of system clock
I20260924 07:18:44.591910 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:45407
--webserver_interface=127.3.246.126
--webserver_port=0
--builtin_ntp_servers=127.3.246.84:39685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:45407
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42803/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 07:18:44.671940 9389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:44.672201 9389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:44.672240 9389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:44.673390 9389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:44.673471 9389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:44.673496 9389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:44.673520 9389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:44.673554 9389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:18:44.673583 9389 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:18:44.673625 9389 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42803/jwks.json
I20260924 07:18:44.675030 9389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:39685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:45407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:45407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42803/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:44.675320 9389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:44.675668 9389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:44.680676 9395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:44.680665 9394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:44.680800 9389 server_base.cc:1034] running on GCE node
W20260924 07:18:44.680665 9397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:44.681154 9389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:44.681452 9389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:44.682593 9389 hybrid_clock.cc:648] HybridClock initialized: now 1790234324682581 us; error 35 us; skew 500 ppm
I20260924 07:18:44.683861 9389 webserver.cc:460] Webserver started at http://127.3.246.126:41601/ using document root <none> and password file <none>
I20260924 07:18:44.684111 9389 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:44.684158 9389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:44.684289 9389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:44.685352 9389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "51e8f81c7d8748039bc9db3a7d17f752"
format_stamp: "Formatted at 2026-09-24 07:18:44 on dist-test-slave-rc1w"
server_key: "14c27bdfd3bbd91f10f974dcdc7063fc"
server_key_iv: "70c827ca17435364c87a4a89fddab6b7"
server_key_version: "encryptionkey@0"
I20260924 07:18:44.685788 9389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "51e8f81c7d8748039bc9db3a7d17f752"
format_stamp: "Formatted at 2026-09-24 07:18:44 on dist-test-slave-rc1w"
server_key: "14c27bdfd3bbd91f10f974dcdc7063fc"
server_key_iv: "70c827ca17435364c87a4a89fddab6b7"
server_key_version: "encryptionkey@0"
I20260924 07:18:44.687124 9389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:44.687867 9403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:44.688073 9389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:44.688170 9389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "51e8f81c7d8748039bc9db3a7d17f752"
format_stamp: "Formatted at 2026-09-24 07:18:44 on dist-test-slave-rc1w"
server_key: "14c27bdfd3bbd91f10f974dcdc7063fc"
server_key_iv: "70c827ca17435364c87a4a89fddab6b7"
server_key_version: "encryptionkey@0"
I20260924 07:18:44.688251 9389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:44.696180 9389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:44.696466 9389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:44.696614 9389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:44.700444 9389 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:45407
I20260924 07:18:44.700474 9455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:45407 every 8 connection(s)
I20260924 07:18:44.700809 9389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:44.701414 9456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:44.703975 9456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752: Bootstrap starting.
I20260924 07:18:44.704474 9456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:44.704696 9456 log.cc:824] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:44.705327 9456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752: No bootstrap required, opened a new log
I20260924 07:18:44.705825 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 9389
I20260924 07:18:44.705945 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:44.706116 4057 external_mini_cluster.cc:1428] Setting key 3ee851f5f991f3353ad35ef6f65a49d6
I20260924 07:18:44.707572 9456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e8f81c7d8748039bc9db3a7d17f752" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 45407 } }
I20260924 07:18:44.707738 9456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:44.707774 9456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e8f81c7d8748039bc9db3a7d17f752, State: Initialized, Role: FOLLOWER
I20260924 07:18:44.707908 9456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e8f81c7d8748039bc9db3a7d17f752" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 45407 } }
I20260924 07:18:44.707988 9456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:44.708017 9456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:44.708055 9456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:44.708740 9456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e8f81c7d8748039bc9db3a7d17f752" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 45407 } }
I20260924 07:18:44.708854 9456 leader_election.cc:302] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51e8f81c7d8748039bc9db3a7d17f752; no voters:
I20260924 07:18:44.708994 9456 leader_election.cc:288] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:44.709069 9461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:44.709203 9461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [term 1 LEADER]: Becoming Leader. State: Replica: 51e8f81c7d8748039bc9db3a7d17f752, State: Running, Role: LEADER
I20260924 07:18:44.709220 9456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:44.709285 9461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e8f81c7d8748039bc9db3a7d17f752" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 45407 } }
I20260924 07:18:44.709610 9461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51e8f81c7d8748039bc9db3a7d17f752. Latest consensus state: current_term: 1 leader_uuid: "51e8f81c7d8748039bc9db3a7d17f752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e8f81c7d8748039bc9db3a7d17f752" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 45407 } } }
I20260924 07:18:44.709622 9462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51e8f81c7d8748039bc9db3a7d17f752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e8f81c7d8748039bc9db3a7d17f752" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 45407 } } }
I20260924 07:18:44.709738 9462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:44.709883 9461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752 [sys.catalog]: This master's current role is: LEADER
W20260924 07:18:44.711032 9477 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:18:44.711086 9477 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:18:44.711133 9468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:44.711265 9468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:44.713048 9468 catalog_manager.cc:1347] Generated new cluster ID: cf60879851d444d5a869b066b8a4767b
I20260924 07:18:44.713100 9468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:44.713727 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.706629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.3.246.126:45407)
0924 07:18:44.706856 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:44.706865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:44.706897 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:18:44.707530 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:44.707539 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:44.707617 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:44.707768 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:44.707980 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.709696 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.710917 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.711470 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.711609 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:44.712286 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:44.712290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:44.712290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:44.712503 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:44.712545 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:44.712558 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:44.712622 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:44.712725 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 07:18:44.712801 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260924 07:18:44.714593 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:18:44.731613 9468 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:44.732237 9468 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:44.736485 9468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51e8f81c7d8748039bc9db3a7d17f752: Generated new TSK 0
I20260924 07:18:44.736688 9468 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:18:44.748958 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.745169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.3.246.126:45407)
0924 07:18:44.745236 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:44.745238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:44.745398 (+ 160us) server_negotiation.cc:407] Connection header received
0924 07:18:44.745478 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:44.745479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:44.745489 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:44.745551 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:18:44.745715 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.746203 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.747360 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.747538 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.747592 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:44.747772 (+ 180us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:18:44.748761 (+ 989us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42803/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 07:18:44.748889 (+ 128us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52364: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42803/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 07:18:44.749019 9460 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52364: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42803/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 07:18:44.749075 9485 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:44.745117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:45407 (local address 127.0.0.1:52364)
0924 07:18:44.745315 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:44.745327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:44.745404 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:44.745567 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:44.745568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:44.745570 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:18:44.745631 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:44.745633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.746282 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.746283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:44.747266 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:44.747268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.747616 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.747617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:44.747677 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:44.747683 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:18:44.748866 (+ 1183us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42803/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 07:18:44.748970 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.246.126:45407: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42803/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":133,"threads_started":1}
I20260924 07:18:44.749286 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 9389
E20260924 07:18:44.759900 9486 webserver.cc:533] Webserver: sslize failed:
E20260924 07:18:44.759970 9486 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T07:18:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (204 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T07:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:44Z Disabled control of system clock
I20260924 07:18:44.795001 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:43775
--webserver_interface=127.3.246.126
--webserver_port=0
--builtin_ntp_servers=127.3.246.84:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:43775
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/testchainca.cert with env {}
W20260924 07:18:44.875106 9490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:44.875304 9490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:44.875326 9490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:44.876402 9490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:44.876438 9490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:44.876451 9490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:44.876462 9490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:44.876474 9490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:18:44.876487 9490 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:18:44.876497 9490 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 07:18:44.877928 9490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:43775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:43775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:44.878106 9490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:44.878260 9490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:44.883106 9498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:44.883095 9495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:44.883095 9496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:44.883266 9490 server_base.cc:1034] running on GCE node
I20260924 07:18:44.883556 9490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:44.883885 9490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:44.885073 9490 hybrid_clock.cc:648] HybridClock initialized: now 1790234324885058 us; error 24 us; skew 500 ppm
I20260924 07:18:44.886371 9490 webserver.cc:460] Webserver started at http://127.3.246.126:38937/ using document root <none> and password file <none>
I20260924 07:18:44.886611 9490 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:44.886689 9490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:44.886845 9490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:44.887943 9490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "e7160074184748908008d8592fd00fc1"
format_stamp: "Formatted at 2026-09-24 07:18:44 on dist-test-slave-rc1w"
server_key: "27fd8f12aa2ba8170130985cfdd3b2ae"
server_key_iv: "0ae885e86c442fdd7dbc52614f74cbbe"
server_key_version: "encryptionkey@0"
I20260924 07:18:44.888341 9490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "e7160074184748908008d8592fd00fc1"
format_stamp: "Formatted at 2026-09-24 07:18:44 on dist-test-slave-rc1w"
server_key: "27fd8f12aa2ba8170130985cfdd3b2ae"
server_key_iv: "0ae885e86c442fdd7dbc52614f74cbbe"
server_key_version: "encryptionkey@0"
I20260924 07:18:44.889853 9490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:44.890615 9504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:44.890826 9490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:44.890925 9490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "e7160074184748908008d8592fd00fc1"
format_stamp: "Formatted at 2026-09-24 07:18:44 on dist-test-slave-rc1w"
server_key: "27fd8f12aa2ba8170130985cfdd3b2ae"
server_key_iv: "0ae885e86c442fdd7dbc52614f74cbbe"
server_key_version: "encryptionkey@0"
I20260924 07:18:44.891012 9490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:44.903728 9490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:44.904006 9490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:44.904156 9490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:44.908407 9490 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:43775
I20260924 07:18:44.908442 9556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:43775 every 8 connection(s)
I20260924 07:18:44.908780 9490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:44.909029 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 9490
I20260924 07:18:44.909140 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:44.909363 4057 external_mini_cluster.cc:1428] Setting key 0dd7a5388001823d2b1ab276d7f99884
I20260924 07:18:44.910022 9557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:44.913002 9557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1: Bootstrap starting.
I20260924 07:18:44.913738 9557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:44.914069 9557 log.cc:824] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:44.914816 9557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1: No bootstrap required, opened a new log
I20260924 07:18:44.915004 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.909884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.3.246.126:43775)
0924 07:18:44.910094 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:44.910103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:44.910136 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:18:44.910590 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:44.910596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:44.910654 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:44.910776 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:44.910996 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.911849 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.912961 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.913422 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.913518 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:44.914268 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:44.914273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:44.914273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:18:44.914440 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:44.914471 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:18:44.914481 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:44.914535 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:44.914622 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 07:18:44.914710 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":110,"threads_started":1}
I20260924 07:18:44.915912 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:18:44.917083 9557 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7160074184748908008d8592fd00fc1" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43775 } }
I20260924 07:18:44.917213 9557 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:44.917236 9557 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7160074184748908008d8592fd00fc1, State: Initialized, Role: FOLLOWER
I20260924 07:18:44.917377 9557 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7160074184748908008d8592fd00fc1" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43775 } }
I20260924 07:18:44.917452 9557 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:44.917493 9557 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:44.917543 9557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:44.918169 9557 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7160074184748908008d8592fd00fc1" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43775 } }
I20260924 07:18:44.918284 9557 leader_election.cc:302] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7160074184748908008d8592fd00fc1; no voters:
I20260924 07:18:44.918440 9557 leader_election.cc:288] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:44.918510 9562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:44.918654 9562 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [term 1 LEADER]: Becoming Leader. State: Replica: e7160074184748908008d8592fd00fc1, State: Running, Role: LEADER
I20260924 07:18:44.918707 9557 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:44.918753 9562 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7160074184748908008d8592fd00fc1" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43775 } }
I20260924 07:18:44.919106 9563 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7160074184748908008d8592fd00fc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7160074184748908008d8592fd00fc1" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43775 } } }
I20260924 07:18:44.919111 9564 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7160074184748908008d8592fd00fc1. Latest consensus state: current_term: 1 leader_uuid: "e7160074184748908008d8592fd00fc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7160074184748908008d8592fd00fc1" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43775 } } }
I20260924 07:18:44.919234 9563 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:44.919253 9564 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:44.919749 9571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:44.920251 9571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:44.921772 9571 catalog_manager.cc:1347] Generated new cluster ID: 6ee262d864354002a40e4a95c73e65ce
I20260924 07:18:44.921847 9571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:44.938104 9571 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:44.938668 9571 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:44.944468 9571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7160074184748908008d8592fd00fc1: Generated new TSK 0
I20260924 07:18:44.944666 9571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:18:45.022776 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:44.961304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.3.246.126:43775)
0924 07:18:44.961370 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:44.961373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:44.961472 (+ 99us) server_negotiation.cc:407] Connection header received
0924 07:18:44.961531 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:44.961532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:44.961546 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:44.961634 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:18:44.961866 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.962647 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.963466 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.963787 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.963859 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:44.964047 (+ 188us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:18:45.022469 (+ 58422us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 07:18:45.022691 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55058: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 07:18:45.022861 9560 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55058: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 07:18:45.022873 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:44.961208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43775 (local address 127.0.0.1:55058)
0924 07:18:44.961402 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:44.961412 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:44.961469 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:44.961719 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:44.961720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:44.961723 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:18:44.961788 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:44.961790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.962775 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.962776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:44.963384 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:44.963386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:44.963889 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:44.963891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:44.963939 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:44.963944 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:18:45.022631 (+ 58687us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 07:18:45.022758 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.246.126:43775: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":115,"threads_started":1}
I20260924 07:18:45.023376 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 9490
2026-09-24T07:18:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (253 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (88 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:45 dist-test-slave-rc1w krb5kdc[9598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:45 dist-test-slave-rc1w krb5kdc[9598](info): set up 2 sockets
Sep 24 07:18:45 dist-test-slave-rc1w krb5kdc[9598](info): commencing operation
krb5kdc: starting...
W20260924 07:18:46.233995 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:46 dist-test-slave-rc1w krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:18:46.357905 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:33433
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:33433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:46.441282 9611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:46.441550 9611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:46.441591 9611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:46.442831 9611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:46.442907 9611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:46.442937 9611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:46.442974 9611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:46.443001 9611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:46.444461 9611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:33433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:33433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:46.444761 9611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:46.444996 9611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:46.450219 9619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:46.450230 9616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:46.450242 9617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:46.450280 9611 server_base.cc:1034] running on GCE node
I20260924 07:18:46.450805 9611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:46.451140 9611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:46.452345 9611 hybrid_clock.cc:648] HybridClock initialized: now 1790234326452323 us; error 43 us; skew 500 ppm
Sep 24 07:18:46 dist-test-slave-rc1w krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:46.463975 9611 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:18:46.464486 9611 webserver.cc:460] Webserver started at http://127.3.246.126:45867/ using document root <none> and password file <none>
I20260924 07:18:46.464810 9611 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:46.464918 9611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:46.465085 9611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:46.466326 9611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "42f82534df9f41e3a99a06e35122a4bf"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-rc1w"
server_key: "99be665ee3c5de93e062028bfa1e7abe"
server_key_iv: "e591479b09d90bc129daaf0b114f018e"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.466717 9611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "42f82534df9f41e3a99a06e35122a4bf"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-rc1w"
server_key: "99be665ee3c5de93e062028bfa1e7abe"
server_key_iv: "e591479b09d90bc129daaf0b114f018e"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.468178 9611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:46.468981 9626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:46.469162 9611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:46.469273 9611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "42f82534df9f41e3a99a06e35122a4bf"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-rc1w"
server_key: "99be665ee3c5de93e062028bfa1e7abe"
server_key_iv: "e591479b09d90bc129daaf0b114f018e"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.469365 9611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:46.493424 9611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:46.496516 9611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:46.496731 9611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:46.501901 9611 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:33433
I20260924 07:18:46.501969 9678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:33433 every 8 connection(s)
I20260924 07:18:46.502399 9611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:46.503171 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 9611
I20260924 07:18:46.503298 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:46.503484 9679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:46.503502 4057 external_mini_cluster.cc:1428] Setting key b3944c74c9eff4b9ca4828a1d0345094
I20260924 07:18:46.506916 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf: Bootstrap starting.
I20260924 07:18:46.507613 9679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:46.507906 9679 log.cc:824] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:46.508589 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf: No bootstrap required, opened a new log
I20260924 07:18:46.510824 9679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f82534df9f41e3a99a06e35122a4bf" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33433 } }
I20260924 07:18:46.511000 9679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 24 07:18:46 dist-test-slave-rc1w krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:46.511035 9679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f82534df9f41e3a99a06e35122a4bf, State: Initialized, Role: FOLLOWER
I20260924 07:18:46.511175 9679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f82534df9f41e3a99a06e35122a4bf" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33433 } }
I20260924 07:18:46.511241 9679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:46.511263 9679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:46.511335 9679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:46.512018 9679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f82534df9f41e3a99a06e35122a4bf" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33433 } }
I20260924 07:18:46.512138 9679 leader_election.cc:302] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42f82534df9f41e3a99a06e35122a4bf; no voters:
I20260924 07:18:46.512306 9679 leader_election.cc:288] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:46.512339 9684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:46.512528 9684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [term 1 LEADER]: Becoming Leader. State: Replica: 42f82534df9f41e3a99a06e35122a4bf, State: Running, Role: LEADER
I20260924 07:18:46.512583 9679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:46.512701 9684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f82534df9f41e3a99a06e35122a4bf" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33433 } }
I20260924 07:18:46.513074 9684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42f82534df9f41e3a99a06e35122a4bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f82534df9f41e3a99a06e35122a4bf" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33433 } } }
I20260924 07:18:46.513217 9684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:46.513402 9684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42f82534df9f41e3a99a06e35122a4bf. Latest consensus state: current_term: 1 leader_uuid: "42f82534df9f41e3a99a06e35122a4bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f82534df9f41e3a99a06e35122a4bf" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33433 } } }
I20260924 07:18:46.513506 9684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf [sys.catalog]: This master's current role is: LEADER
W20260924 07:18:46.514587 9699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:18:46.514657 9699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:18:46.514717 9693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:46.514909 9693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:46.515982 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.504207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.3.246.126:33433)
0924 07:18:46.504373 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.504383 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.504419 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:18:46.504906 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.504913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.504961 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.505124 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.505956 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.507340 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.508371 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.508832 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.508962 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.511619 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.511624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.511625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.511640 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.514233 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.514667 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.514670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.514670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.514776 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.514997 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.514997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.514998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.515325 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.515511 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.515650 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.515792 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260924 07:18:46.516868 9693 catalog_manager.cc:1347] Generated new cluster ID: 7dce14308f18449b8533e8d441c614a5
I20260924 07:18:46.516923 9693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:46.523706 9693 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:46.524406 9693 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:46.535845 9693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42f82534df9f41e3a99a06e35122a4bf: Generated new TSK 0
I20260924 07:18:46.536049 9693 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:18:46.571147 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:33433
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:46.672680 9705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:46.672876 9705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:46.672901 9705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:46.674247 9705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:46.674290 9705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:46.674320 9705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:46.675815 9705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:33433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:46.676064 9705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:46.676329 9705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:46.681346 9711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:46.681318 9713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:46.681317 9710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:46.681697 9705 server_base.cc:1034] running on GCE node
I20260924 07:18:46.681911 9705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:46.682111 9705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:46.683276 9705 hybrid_clock.cc:648] HybridClock initialized: now 1790234326683270 us; error 40 us; skew 500 ppm
Sep 24 07:18:46 dist-test-slave-rc1w krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:46.685763 9705 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:18:46.686206 9705 webserver.cc:460] Webserver started at http://127.3.246.65:44785/ using document root <none> and password file <none>
I20260924 07:18:46.686439 9705 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:46.686482 9705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:46.686569 9705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:46.687530 9705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "fa4fbf5661724c13a976c663f5f91ae6"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-rc1w"
server_key: "7beb0eb38de21060c6bc7a52d7662a78"
server_key_iv: "74d74c5bb7c98947f73d02c33f7259c5"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.687816 9705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "fa4fbf5661724c13a976c663f5f91ae6"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-rc1w"
server_key: "7beb0eb38de21060c6bc7a52d7662a78"
server_key_iv: "74d74c5bb7c98947f73d02c33f7259c5"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.688997 9705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:18:46.689639 9720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:46.689869 9705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:46.689963 9705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "fa4fbf5661724c13a976c663f5f91ae6"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-rc1w"
server_key: "7beb0eb38de21060c6bc7a52d7662a78"
server_key_iv: "74d74c5bb7c98947f73d02c33f7259c5"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.690047 9705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:46.713161 9705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:46.716058 9705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:46.716253 9705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:46.716488 9705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:46.716866 9705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:46.716940 9705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:46.716979 9705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:46.717013 9705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:46.724493 9705 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:33773
I20260924 07:18:46.724520 9833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:33773 every 8 connection(s)
I20260924 07:18:46.724922 9705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:46.726811 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 9705
I20260924 07:18:46.726930 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:46.727142 4057 external_mini_cluster.cc:1428] Setting key 51c12499a7c83a4aec965078fd4c0052
Sep 24 07:18:46 dist-test-slave-rc1w krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:46.735597 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.725799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:48319 (local address 127.3.246.126:33433)
0924 07:18:46.726343 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.726345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.726357 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:18:46.727425 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.727426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.727440 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.727490 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.728246 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.728872 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.729961 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.730178 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.730254 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.732502 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.732503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.732504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.732513 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.734212 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.734601 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.734602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.734602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.734674 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.734909 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.734910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.734910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.735052 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.735125 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.735388 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.735489 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513}
I20260924 07:18:46.735622 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.725831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:33433 (local address 127.3.246.65:48319)
0924 07:18:46.726006 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.726040 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:46.727316 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:46.727558 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:46.727562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:46.727859 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:46.728136 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.728148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.728969 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.728970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.729870 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.729873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.730260 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.730261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.730397 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.730916 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:46.730925 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:46.732410 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:46.734291 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.734297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.734299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.734522 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.734745 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.734745 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.734746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.734839 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.735124 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:46.735128 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:46.735237 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 07:18:46.735314 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:18:46.735458 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260924 07:18:46.736289 9834 heartbeater.cc:347] Connected to a master server at 127.3.246.126:33433
I20260924 07:18:46.736413 9834 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:46.736591 9834 heartbeater.cc:511] Master 127.3.246.126:33433 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
I20260924 07:18:46.737151 9643 ts_manager.cc:194] Registered new tserver with Master: fa4fbf5661724c13a976c663f5f91ae6 (127.3.246.65:33773)
I20260924 07:18:46.737834 9643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:48319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:18:46.753201 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:33433
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:46.835201 9839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:46.835429 9839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:46.835453 9839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:46.836524 9839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:46.836555 9839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:46.836578 9839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:46.838043 9839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:33433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:46.838223 9839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:46.838393 9839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:46.843231 9844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:46.843278 9847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:46.843247 9845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:46.843528 9839 server_base.cc:1034] running on GCE node
I20260924 07:18:46.843770 9839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:46.844003 9839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:46.845171 9839 hybrid_clock.cc:648] HybridClock initialized: now 1790234326845162 us; error 37 us; skew 500 ppm
Sep 24 07:18:46 dist-test-slave-rc1w krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:46.847801 9839 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:18:46.848219 9839 webserver.cc:460] Webserver started at http://127.3.246.66:37981/ using document root <none> and password file <none>
I20260924 07:18:46.848451 9839 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:46.848516 9839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:46.848644 9839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:46.849736 9839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "7841eadb88824d3e8bdedc68e844d889"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-rc1w"
server_key: "182d1cf131ab403083389061b746dd22"
server_key_iv: "bf1e69e35427f93beb6f57a229366ee8"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.850102 9839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "7841eadb88824d3e8bdedc68e844d889"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-rc1w"
server_key: "182d1cf131ab403083389061b746dd22"
server_key_iv: "bf1e69e35427f93beb6f57a229366ee8"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.851399 9839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:46.852187 9854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:46.852397 9839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:46.852502 9839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "7841eadb88824d3e8bdedc68e844d889"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-rc1w"
server_key: "182d1cf131ab403083389061b746dd22"
server_key_iv: "bf1e69e35427f93beb6f57a229366ee8"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.852591 9839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:46.860880 9839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:46.862793 9839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:46.862949 9839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:46.863161 9839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:46.863454 9839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:46.863505 9839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:46.863545 9839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:46.863574 9839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:46.869242 9839 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:44941
I20260924 07:18:46.869292 9967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:44941 every 8 connection(s)
I20260924 07:18:46.869663 9839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:18:46 dist-test-slave-rc1w krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:46.877982 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 9839
I20260924 07:18:46.878111 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:46.878314 4057 external_mini_cluster.cc:1428] Setting key 320736db1b816a1aa912ba4b9d6cf708
I20260924 07:18:46.879366 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.870490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:44771 (local address 127.3.246.126:33433)
0924 07:18:46.870586 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.870588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.870777 (+ 189us) server_negotiation.cc:407] Connection header received
0924 07:18:46.871753 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.871754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.871766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.871848 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.872578 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.873080 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.874169 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.874399 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.874495 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.876418 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.876419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.876420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.876427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.877835 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.878309 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.878311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.878311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.878381 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.878623 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.878624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.878624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.878749 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.878810 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.879129 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.879220 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 07:18:46.879374 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.870408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:33433 (local address 127.3.246.66:44771)
0924 07:18:46.870630 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.870665 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:46.871622 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:46.871900 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:46.871904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:46.872255 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:46.872487 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.872492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.873178 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.873180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.874085 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.874088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.874506 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.874507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.874625 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.875042 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:46.875050 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:46.876328 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:46.877926 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.877932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.877934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.878221 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.878435 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.878436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.878436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.878567 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.878816 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:46.878822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:46.878999 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 07:18:46.879091 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 07:18:46.879223 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
I20260924 07:18:46.880052 9968 heartbeater.cc:347] Connected to a master server at 127.3.246.126:33433
I20260924 07:18:46.880167 9968 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:46.880384 9968 heartbeater.cc:511] Master 127.3.246.126:33433 requested a full tablet report, sending...
I20260924 07:18:46.880831 9643 ts_manager.cc:194] Registered new tserver with Master: 7841eadb88824d3e8bdedc68e844d889 (127.3.246.66:44941)
I20260924 07:18:46.881191 9643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:44771
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:18:46.904739 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:33433
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:46.986035 9973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:46.986228 9973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:46.986251 9973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:46.987452 9973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:46.987491 9973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:46.987517 9973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:46.989001 9973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:33433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:46.989233 9973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:46.989470 9973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:46.994364 9981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:46.994400 9979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:46.994427 9978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:46.994623 9973 server_base.cc:1034] running on GCE node
I20260924 07:18:46.994851 9973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:46.995085 9973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:46.996259 9973 hybrid_clock.cc:648] HybridClock initialized: now 1790234326996246 us; error 38 us; skew 500 ppm
Sep 24 07:18:46 dist-test-slave-rc1w krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:46.999095 9973 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:18:46.999524 9973 webserver.cc:460] Webserver started at http://127.3.246.67:34211/ using document root <none> and password file <none>
I20260924 07:18:46.999774 9973 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:46.999842 9973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:46.999997 9973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:47.001027 9973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "f9df6a37ebb244cdafa3c29c3d33d6ce"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-rc1w"
server_key: "20774cdd24803c4c54d679934ba9c5e3"
server_key_iv: "9de790fcdecea1cf9ced8a92816e6c0a"
server_key_version: "encryptionkey@0"
I20260924 07:18:47.001381 9973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "f9df6a37ebb244cdafa3c29c3d33d6ce"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-rc1w"
server_key: "20774cdd24803c4c54d679934ba9c5e3"
server_key_iv: "9de790fcdecea1cf9ced8a92816e6c0a"
server_key_version: "encryptionkey@0"
I20260924 07:18:47.002933 9973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:18:47.003925 9988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:47.004144 9973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:47.004263 9973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "f9df6a37ebb244cdafa3c29c3d33d6ce"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-rc1w"
server_key: "20774cdd24803c4c54d679934ba9c5e3"
server_key_iv: "9de790fcdecea1cf9ced8a92816e6c0a"
server_key_version: "encryptionkey@0"
I20260924 07:18:47.004382 9973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:47.021662 9973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:47.023842 9973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:47.024021 9973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:47.024272 9973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:47.024580 9973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:47.024645 9973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:47.024712 9973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:47.024744 9973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:47.030781 9973 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:38131
I20260924 07:18:47.030819 10101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:38131 every 8 connection(s)
I20260924 07:18:47.031147 9973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:47.039613 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 9973
I20260924 07:18:47.039703 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:47.039894 4057 external_mini_cluster.cc:1428] Setting key 0a5d66f70eaa16667efc53b96183efc9
I20260924 07:18:47.040373 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:47.031895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:40505 (local address 127.3.246.126:33433)
0924 07:18:47.031942 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:47.031944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:47.032109 (+ 165us) server_negotiation.cc:407] Connection header received
0924 07:18:47.033042 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:47.033043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:47.033054 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:47.033110 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:47.033770 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:47.034283 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:47.035245 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:47.035445 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:47.035498 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:47.037650 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:47.037651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:47.037651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:47.037658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:47.038919 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:47.039293 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:47.039294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:47.039294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:47.039350 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:47.039568 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:47.039569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:47.039570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:47.039721 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:47.039790 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:47.040172 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 07:18:47.040283 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:18:47.040512 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:47.031861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:33433 (local address 127.3.246.67:40505)
0924 07:18:47.032009 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:47.032043 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:47.032938 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:47.033161 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:47.033165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:47.033415 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:47.033603 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:47.033607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:47.034374 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:47.034374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:47.035167 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:47.035169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:47.035535 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:47.035536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:47.035639 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:47.036205 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:47.036213 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:47.037571 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:47.039001 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:47.039008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:47.039010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:47.039215 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:47.039408 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:47.039409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:47.039409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:47.039501 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:47.039816 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:47.039822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:47.039999 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 07:18:47.040131 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 07:18:47.040287 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260924 07:18:47.041050 10102 heartbeater.cc:347] Connected to a master server at 127.3.246.126:33433
I20260924 07:18:47.041132 10102 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:47.041275 10102 heartbeater.cc:511] Master 127.3.246.126:33433 requested a full tablet report, sending...
I20260924 07:18:47.041613 9643 ts_manager.cc:194] Registered new tserver with Master: f9df6a37ebb244cdafa3c29c3d33d6ce (127.3.246.67:38131)
I20260924 07:18:47.042006 9643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:40505
I20260924 07:18:47.051680 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/test-admin.keytab.
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:47.066267 4057 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 07:18:47.068053 10109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:47.067156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.3.246.65:33773)
0924 07:18:47.067347 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:47.067351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:47.067366 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:18:47.067444 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:47.067447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:47.067462 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:47.067534 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:47.067954 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40852: BlockingRecv error: recv got EOF from 127.0.0.1:40852 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
W20260924 07:18:47.068095 4057 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/krb5cc' of type FILE
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:47.069079 4057 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:47.069965 4057 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.65@KRBTEST.COM
I20260924 07:18:47.077942 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:47.070478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.3.246.65:33773)
0924 07:18:47.070541 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:47.070543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:47.070601 (+ 58us) server_negotiation.cc:407] Connection header received
0924 07:18:47.070716 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:47.070717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:47.070730 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:47.070800 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:47.071182 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:47.071890 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:47.072779 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:47.073099 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:47.073198 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:47.075315 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:47.075322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:47.075322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:47.075329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:47.076766 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:47.077126 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:47.077129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:47.077129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:47.077194 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:47.077397 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:47.077398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:47.077398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:47.077519 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:47.077620 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:47.077809 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 07:18:47.077874 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 07:18:47.079695 10109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:47.079173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.3.246.65:33773)
0924 07:18:47.079224 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:47.079225 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:47.079262 (+ 37us) server_negotiation.cc:407] Connection header received
0924 07:18:47.079329 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:47.079329 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:47.079337 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:47.079379 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:47.079650 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40878: BlockingRecv error: recv got EOF from 127.0.0.1:40878 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 07:18:47.079833 4057 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/krb5cc)
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:47.080749 4057 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:47.081646 4057 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.65@KRBTEST.COM
I20260924 07:18:47.088621 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:47.082194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.3.246.65:33773)
0924 07:18:47.082241 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:47.082243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:47.082277 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:18:47.082360 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:47.082361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:47.082368 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:47.082427 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:47.082830 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:47.083293 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:47.083951 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:47.084148 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:47.084204 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:47.086341 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:47.086343 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:47.086343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:47.086350 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:47.087550 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:47.087883 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:47.087885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:47.087885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:47.087941 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:47.088137 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:47.088138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:47.088138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:47.088244 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:47.088342 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:47.088459 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 07:18:47.088527 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 07:18:47.090198 10109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:47.089712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.3.246.65:33773)
0924 07:18:47.089745 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:47.089747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:47.089811 (+ 64us) server_negotiation.cc:407] Connection header received
0924 07:18:47.089840 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:47.089841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:47.089847 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:47.089893 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:47.090121 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40902: BlockingRecv error: recv got EOF from 127.0.0.1:40902 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 07:18:47.090289 4057 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234301618289-4057-0/krb5kdc/krb5cc' of type FILE
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:47.091241 4057 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:47.092203 4057 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.65@KRBTEST.COM
I20260924 07:18:47.099565 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:47.092745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.3.246.65:33773)
0924 07:18:47.092798 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:47.092800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:47.092866 (+ 66us) server_negotiation.cc:407] Connection header received
0924 07:18:47.092939 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:47.092940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:47.092949 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:47.093003 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:47.093400 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:47.093951 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:47.094798 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:47.094987 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:47.095069 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:47.097226 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:47.097229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:47.097230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:47.097240 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:47.098454 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:47.098816 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:47.098817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:47.098817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:47.098876 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:47.099073 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:47.099074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:47.099074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:47.099173 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:47.099287 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:47.099395 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 07:18:47.099465 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:18:47.100219 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 9705
I20260924 07:18:47.104866 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 9839
I20260924 07:18:47.109243 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 9973
I20260924 07:18:47.113617 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 9611
2026-09-24T07:18:47Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1914 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[10122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[10122](info): set up 2 sockets
Sep 24 07:18:47 dist-test-slave-rc1w krb5kdc[10122](info): commencing operation
krb5kdc: starting...
W20260924 07:18:48.154089 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:48 dist-test-slave-rc1w krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.126.keytab.
Entry for principal oryx/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.126.keytab.
I20260924 07:18:48.273520 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:46187
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.126.keytab
--principal=oryx/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:46187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:48.353536 10135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:48.353794 10135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:48.353833 10135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:48.354974 10135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:48.355043 10135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:48.355070 10135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:48.355103 10135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:48.355126 10135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:48.356630 10135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:46187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.126.keytab
--principal=oryx/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:46187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:48.356905 10135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:48.357146 10135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:48.362040 10140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:48.362005 10143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:48.362021 10141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:48.362258 10135 server_base.cc:1034] running on GCE node
I20260924 07:18:48.362512 10135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:48.362818 10135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:48.363970 10135 hybrid_clock.cc:648] HybridClock initialized: now 1790234328363960 us; error 33 us; skew 500 ppm
Sep 24 07:18:48 dist-test-slave-rc1w krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234328, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:48.366629 10135 init.cc:377] Logged in from keytab as oryx/127.3.246.126@KRBTEST.COM (short username oryx)
I20260924 07:18:48.367092 10135 webserver.cc:460] Webserver started at http://127.3.246.126:41081/ using document root <none> and password file <none>
I20260924 07:18:48.367339 10135 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:48.367408 10135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:48.367551 10135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:48.368599 10135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "2744c465ca96b6f363e5685c8da71542"
server_key_iv: "abc6852fa1d376664bb8d434302c7e40"
server_key_version: "encryptionkey@0"
I20260924 07:18:48.368952 10135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "2744c465ca96b6f363e5685c8da71542"
server_key_iv: "abc6852fa1d376664bb8d434302c7e40"
server_key_version: "encryptionkey@0"
I20260924 07:18:48.370360 10135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:18:48.371122 10150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:48.371357 10135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:48.371466 10135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "2744c465ca96b6f363e5685c8da71542"
server_key_iv: "abc6852fa1d376664bb8d434302c7e40"
server_key_version: "encryptionkey@0"
I20260924 07:18:48.371541 10135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:48.386307 10135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:48.388675 10135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:48.388851 10135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:48.392817 10135 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:46187
I20260924 07:18:48.392853 10202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:46187 every 8 connection(s)
I20260924 07:18:48.393247 10135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:48.394304 10203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:48.396514 10203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab: Bootstrap starting.
I20260924 07:18:48.397097 10203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:48.397408 10203 log.cc:824] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:48.398159 10203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab: No bootstrap required, opened a new log
I20260924 07:18:48.398252 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 10135
I20260924 07:18:48.398367 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:48.398563 4057 external_mini_cluster.cc:1428] Setting key 0d6eee4fe0bc9cd949cf4276a78d3f68
I20260924 07:18:48.400142 10203 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } }
I20260924 07:18:48.400305 10203 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:48.400343 10203 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f9ee1f206c43caa1a7f55aae00a8ab, State: Initialized, Role: FOLLOWER
I20260924 07:18:48.400502 10203 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } }
I20260924 07:18:48.400596 10203 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:48.400642 10203 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:48.400697 10203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:48.401736 10203 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } }
I20260924 07:18:48.401832 10203 leader_election.cc:302] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4f9ee1f206c43caa1a7f55aae00a8ab; no voters:
I20260924 07:18:48.402000 10203 leader_election.cc:288] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:48.402045 10208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:48.402179 10208 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [term 1 LEADER]: Becoming Leader. State: Replica: e4f9ee1f206c43caa1a7f55aae00a8ab, State: Running, Role: LEADER
I20260924 07:18:48.402269 10203 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:48.402266 10208 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } }
I20260924 07:18:48.403685 10209 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } }
I20260924 07:18:48.403954 10209 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:48.403685 10210 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4f9ee1f206c43caa1a7f55aae00a8ab. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } }
I20260924 07:18:48.404122 10210 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:48.404439 10223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:48.404954 10223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 07:18:48 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:48.406613 10223 catalog_manager.cc:1347] Generated new cluster ID: f5a9a963570944cc95a964bb074d0315
I20260924 07:18:48.406679 10223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:48.410898 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:48.399119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.3.246.126:46187)
0924 07:18:48.399311 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:48.399320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:48.399353 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:18:48.400124 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:48.400129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:48.400182 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:48.400292 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:48.401062 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.402204 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.403340 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.403987 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.404132 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:48.406823 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:48.406826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:48.406827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:48.406838 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:48.408715 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:48.409069 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:48.409072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:48.409072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:48.409203 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:48.409379 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:48.409379 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:48.409380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:48.409641 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:48.410168 (+ 527us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:48.410574 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 07:18:48.410706 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260924 07:18:48.427121 10223 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:48.427632 10223 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:48.434558 10223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab: Generated new TSK 0
I20260924 07:18:48.434762 10223 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab.
Entry for principal oryx/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab.
I20260924 07:18:48.468819 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46187
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab
--principal=oryx/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:48.555224 10229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:48.555430 10229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:48.555454 10229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:48.556589 10229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:48.556630 10229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:48.556685 10229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:48.558133 10229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab
--principal=oryx/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:48.558372 10229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:48.558616 10229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:48.563694 10234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:48.563773 10229 server_base.cc:1034] running on GCE node
W20260924 07:18:48.563709 10235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:48.563665 10237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:48.564077 10229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:48.564324 10229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:48.565485 10229 hybrid_clock.cc:648] HybridClock initialized: now 1790234328565478 us; error 33 us; skew 500 ppm
Sep 24 07:18:48 dist-test-slave-rc1w krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234328, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:48.568003 10229 init.cc:377] Logged in from keytab as oryx/127.3.246.65@KRBTEST.COM (short username oryx)
I20260924 07:18:48.568459 10229 webserver.cc:460] Webserver started at http://127.3.246.65:42913/ using document root <none> and password file <none>
I20260924 07:18:48.568683 10229 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:48.568751 10229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:48.568903 10229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:48.570009 10229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "7be36e4202744663900a72eb10a5ae86"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "2dde677999a5efd3f2c58fe105e38c5a"
server_key_iv: "4dbdb138342c293570ae5b8f0c2b0ec5"
server_key_version: "encryptionkey@0"
I20260924 07:18:48.570371 10229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "7be36e4202744663900a72eb10a5ae86"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "2dde677999a5efd3f2c58fe105e38c5a"
server_key_iv: "4dbdb138342c293570ae5b8f0c2b0ec5"
server_key_version: "encryptionkey@0"
I20260924 07:18:48.571662 10229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:48.572409 10244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:48.572628 10229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:48.572734 10229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "7be36e4202744663900a72eb10a5ae86"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "2dde677999a5efd3f2c58fe105e38c5a"
server_key_iv: "4dbdb138342c293570ae5b8f0c2b0ec5"
server_key_version: "encryptionkey@0"
I20260924 07:18:48.572813 10229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:48.581936 10229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:48.583948 10229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:48.584117 10229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:48.584367 10229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:48.584690 10229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:48.584748 10229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:48.584791 10229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:48.584815 10229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:48.590806 10229 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:41413
I20260924 07:18:48.590864 10357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:41413 every 8 connection(s)
I20260924 07:18:48.591220 10229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:48.593611 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 10229
I20260924 07:18:48.593797 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:48.594022 4057 external_mini_cluster.cc:1428] Setting key 07f44d53b38fc5f9d8efa5cb2fc9a670
Sep 24 07:18:48 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234328, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:48.602097 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:48.591987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:54541 (local address 127.3.246.126:46187)
0924 07:18:48.592067 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:48.592070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:48.592301 (+ 231us) server_negotiation.cc:407] Connection header received
0924 07:18:48.592761 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:48.592763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:48.592778 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:48.592831 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:48.593902 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.594635 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.595646 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.595998 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.596072 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:48.598933 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:48.598935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:48.598935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:48.598943 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:48.600452 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:48.600969 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:48.600970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:48.600970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:48.601030 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:48.601356 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:48.601357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:48.601358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:48.601477 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:48.601544 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:48.601959 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 07:18:48.602032 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:18:48.602401 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:48.591928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.65:54541)
0924 07:18:48.592166 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:48.592203 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:48.592672 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:48.592886 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:48.592889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:48.593474 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:48.593661 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:48.593815 (+ 154us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.594724 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.594725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:48.595549 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:48.595551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.596089 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.596090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:48.596245 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:48.596823 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:48.596834 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:48.598799 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:48.600541 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:48.600549 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:48.600552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:48.600872 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:48.601110 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:48.601111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:48.601111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:48.601258 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:48.601559 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:48.601567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:48.601797 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 07:18:48.601924 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 07:18:48.602105 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":133,"threads_started":1}
WARNING: no policy specified for oryx/127.3.246.66@KRBTEST.COM; defaulting to no policy
I20260924 07:18:48.602990 10358 heartbeater.cc:347] Connected to a master server at 127.3.246.126:46187
I20260924 07:18:48.603104 10358 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:48.603281 10358 heartbeater.cc:511] Master 127.3.246.126:46187 requested a full tablet report, sending...
I20260924 07:18:48.603811 10167 ts_manager.cc:194] Registered new tserver with Master: 7be36e4202744663900a72eb10a5ae86 (127.3.246.65:41413)
I20260924 07:18:48.604471 10167 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:54541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab.
Entry for principal oryx/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab.
I20260924 07:18:48.621236 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46187
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab
--principal=oryx/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:48.702587 10363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:48.702782 10363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:48.702829 10363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:48.704104 10363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:48.704144 10363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:48.704170 10363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:48.705622 10363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab
--principal=oryx/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:48.705871 10363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:48.706094 10363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:48.710947 10368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:48.711040 10363 server_base.cc:1034] running on GCE node
W20260924 07:18:48.710913 10369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:48.711139 10371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:48.711357 10363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:48.711565 10363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:48.712729 10363 hybrid_clock.cc:648] HybridClock initialized: now 1790234328712725 us; error 34 us; skew 500 ppm
Sep 24 07:18:48 dist-test-slave-rc1w krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234328, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:48.715142 10363 init.cc:377] Logged in from keytab as oryx/127.3.246.66@KRBTEST.COM (short username oryx)
I20260924 07:18:48.715641 10363 webserver.cc:460] Webserver started at http://127.3.246.66:45225/ using document root <none> and password file <none>
I20260924 07:18:48.715885 10363 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:48.715950 10363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:48.716104 10363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:48.717130 10363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "878909796545ed548a24feda4177b9fb"
server_key_iv: "02f4703c77f0b9c1afd4c3e72fc40fdc"
server_key_version: "encryptionkey@0"
I20260924 07:18:48.717500 10363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "878909796545ed548a24feda4177b9fb"
server_key_iv: "02f4703c77f0b9c1afd4c3e72fc40fdc"
server_key_version: "encryptionkey@0"
I20260924 07:18:48.718923 10363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:48.719899 10378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:48.720126 10363 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:48.720224 10363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "878909796545ed548a24feda4177b9fb"
server_key_iv: "02f4703c77f0b9c1afd4c3e72fc40fdc"
server_key_version: "encryptionkey@0"
I20260924 07:18:48.720315 10363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:48.731472 10363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:48.733204 10363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:48.733366 10363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:48.733588 10363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:48.733923 10363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:48.733978 10363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:48.734011 10363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:48.734040 10363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:48.739909 10363 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:34417
I20260924 07:18:48.739948 10491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:34417 every 8 connection(s)
I20260924 07:18:48.740259 10363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:48.745903 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 10363
I20260924 07:18:48.746021 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:48.746217 4057 external_mini_cluster.cc:1428] Setting key ada323534f6fc77ea00ed4f06b5d93d1
Sep 24 07:18:48 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234328, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:48.750442 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:48.740993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:45431 (local address 127.3.246.126:46187)
0924 07:18:48.741058 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:48.741060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:48.741238 (+ 178us) server_negotiation.cc:407] Connection header received
0924 07:18:48.741790 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:48.741791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:48.741803 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:48.741873 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:48.742801 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.743360 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.744388 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.744594 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.744683 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:48.747333 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:48.747334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:48.747334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:48.747341 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:48.748854 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:48.749402 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:48.749404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:48.749404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:48.749464 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:48.749803 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:48.749804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:48.749804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:48.749916 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:48.749968 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:48.750308 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 07:18:48.750379 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:18:48.750732 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:48.740972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.66:45431)
0924 07:18:48.741122 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:48.741156 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:48.741656 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:48.741945 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:48.741949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:48.742512 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:48.742693 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:48.742697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.743472 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.743473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:48.744279 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:48.744281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.744694 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.744695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:48.744824 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:48.745365 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:48.745373 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:48.747227 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:48.748945 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:48.748951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:48.748954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:48.749293 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:48.749549 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:48.749550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:48.749551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:48.749723 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:48.749983 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:48.749989 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:48.750153 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 07:18:48.750271 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 07:18:48.750448 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260924 07:18:48.751364 10492 heartbeater.cc:347] Connected to a master server at 127.3.246.126:46187
I20260924 07:18:48.751474 10492 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:48.751677 10492 heartbeater.cc:511] Master 127.3.246.126:46187 requested a full tablet report, sending...
I20260924 07:18:48.752023 10167 ts_manager.cc:194] Registered new tserver with Master: bf13e4ca3f8c4b08bedf0e3654c3e86e (127.3.246.66:34417)
I20260924 07:18:48.752400 10167 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:45431
WARNING: no policy specified for oryx/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab.
Entry for principal oryx/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab.
I20260924 07:18:48.772779 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46187
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab
--principal=oryx/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:48.853904 10497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:48.854106 10497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:48.854130 10497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:48.855298 10497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:48.855341 10497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:48.855386 10497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:48.856848 10497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab
--principal=oryx/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:48.857080 10497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:48.857323 10497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:48.862278 10505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:48.862238 10502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:48.862247 10503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:48.862450 10497 server_base.cc:1034] running on GCE node
I20260924 07:18:48.862635 10497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:48.862849 10497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:48.864027 10497 hybrid_clock.cc:648] HybridClock initialized: now 1790234328864008 us; error 38 us; skew 500 ppm
Sep 24 07:18:48 dist-test-slave-rc1w krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234328, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:48.866465 10497 init.cc:377] Logged in from keytab as oryx/127.3.246.67@KRBTEST.COM (short username oryx)
I20260924 07:18:48.866954 10497 webserver.cc:460] Webserver started at http://127.3.246.67:40789/ using document root <none> and password file <none>
I20260924 07:18:48.867223 10497 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:48.867288 10497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:48.867425 10497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:48.868460 10497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "16eef29dfa904346bcb8ff1485075d58"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "f80b8fe5829636208db4627fe96e173e"
server_key_iv: "535b657beaafd19844eb1512b9e0de74"
server_key_version: "encryptionkey@0"
I20260924 07:18:48.868832 10497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "16eef29dfa904346bcb8ff1485075d58"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "f80b8fe5829636208db4627fe96e173e"
server_key_iv: "535b657beaafd19844eb1512b9e0de74"
server_key_version: "encryptionkey@0"
I20260924 07:18:48.870218 10497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:18:48.871026 10512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:48.871227 10497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:48.871327 10497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "16eef29dfa904346bcb8ff1485075d58"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "f80b8fe5829636208db4627fe96e173e"
server_key_iv: "535b657beaafd19844eb1512b9e0de74"
server_key_version: "encryptionkey@0"
I20260924 07:18:48.871410 10497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:48.884327 10497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:48.886157 10497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:48.886315 10497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:48.886531 10497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:48.886827 10497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:48.886883 10497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:48.886922 10497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:48.886952 10497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:48.892876 10497 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:33321
I20260924 07:18:48.892928 10625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:33321 every 8 connection(s)
I20260924 07:18:48.893244 10497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:48.897660 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 10497
I20260924 07:18:48.897797 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:48.898022 4057 external_mini_cluster.cc:1428] Setting key d221a5cfa8bc1c0aa79e4855c3443d14
Sep 24 07:18:48 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234328, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:48.902307 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:48.894002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:46581 (local address 127.3.246.126:46187)
0924 07:18:48.894072 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:48.894074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:48.894237 (+ 163us) server_negotiation.cc:407] Connection header received
0924 07:18:48.894706 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:48.894707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:48.894730 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:48.894785 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:48.895679 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.896182 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.897156 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.897355 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.897420 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:48.899879 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:48.899880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:48.899880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:48.899886 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:48.901024 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:48.901397 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:48.901398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:48.901398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:48.901447 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:48.901651 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:48.901651 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:48.901651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:48.901813 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:48.901868 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:48.902160 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 07:18:48.902223 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:18:48.902527 10628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:48.893969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.67:46581)
0924 07:18:48.894112 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:48.894152 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:48.894622 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:48.894842 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:48.894845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:48.895408 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:48.895591 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:48.895595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.896260 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.896261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:48.897038 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:48.897040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:48.897436 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:48.897437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:48.897596 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:48.898234 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:48.898243 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:48.899799 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:48.901107 (+ 1308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:48.901113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:48.901114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:48.901327 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:48.901504 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:48.901504 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:48.901505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:48.901597 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:48.901871 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:48.901876 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:48.902042 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 07:18:48.902129 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:18:48.902288 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260924 07:18:48.903038 10626 heartbeater.cc:347] Connected to a master server at 127.3.246.126:46187
I20260924 07:18:48.903131 10626 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:48.903327 10626 heartbeater.cc:511] Master 127.3.246.126:46187 requested a full tablet report, sending...
I20260924 07:18:48.903693 10167 ts_manager.cc:194] Registered new tserver with Master: 16eef29dfa904346bcb8ff1485075d58 (127.3.246.67:33321)
I20260924 07:18:48.904090 10167 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:46581
I20260924 07:18:48.909798 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.246.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.246.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.246.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.125.keytab.
Entry for principal oryx/127.3.246.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.125.keytab.
I20260924 07:18:48.934178 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.125:35077
--webserver_interface=127.3.246.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.125.keytab
--principal=oryx/127.3.246.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:46187,127.3.246.125:35077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:49.014523 10631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:49.014709 10631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:49.014731 10631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:49.015817 10631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:49.015851 10631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:49.015865 10631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:49.015878 10631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:49.015890 10631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:49.017280 10631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:46187,127.3.246.125:35077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.125.keytab
--principal=oryx/127.3.246.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.125:35077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.246.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
W20260924 07:18:49.017377 10631 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.246.126:46187,127.3.246.125:35077'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 07:18:49.017479 10631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:49.017634 10631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:49.022387 10637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.022383 10639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:49.022581 10631 server_base.cc:1034] running on GCE node
W20260924 07:18:49.022651 10636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:49.022930 10631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:49.023232 10631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:49.024389 10631 hybrid_clock.cc:648] HybridClock initialized: now 1790234329024376 us; error 29 us; skew 500 ppm
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:49.026863 10631 init.cc:377] Logged in from keytab as oryx/127.3.246.125@KRBTEST.COM (short username oryx)
I20260924 07:18:49.027236 10631 webserver.cc:460] Webserver started at http://127.3.246.125:37453/ using document root <none> and password file <none>
I20260924 07:18:49.027419 10631 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:49.027458 10631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:49.027544 10631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:49.028484 10631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/data/instance:
uuid: "13af89bf44fa42fe9c5a301d79df7ac2"
format_stamp: "Formatted at 2026-09-24 07:18:49 on dist-test-slave-rc1w"
server_key: "73b90505be868e932bc0319579f5d880"
server_key_iv: "15be502236ae51af34c11dd2a75b26ff"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.028770 10631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/wal/instance:
uuid: "13af89bf44fa42fe9c5a301d79df7ac2"
format_stamp: "Formatted at 2026-09-24 07:18:49 on dist-test-slave-rc1w"
server_key: "73b90505be868e932bc0319579f5d880"
server_key_iv: "15be502236ae51af34c11dd2a75b26ff"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.030009 10631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:49.030671 10646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.030855 10631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:18:49.030943 10631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/wal
uuid: "13af89bf44fa42fe9c5a301d79df7ac2"
format_stamp: "Formatted at 2026-09-24 07:18:49 on dist-test-slave-rc1w"
server_key: "73b90505be868e932bc0319579f5d880"
server_key_iv: "15be502236ae51af34c11dd2a75b26ff"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.031023 10631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:49.044013 10631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:49.046262 10631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:49.046412 10631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:49.050295 10631 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.125:35077
I20260924 07:18:49.050364 10698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.125:35077 every 8 connection(s)
I20260924 07:18:49.050700 10631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/data/info.pb
I20260924 07:18:49.051651 10699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:49.053170 10699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.125@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:49.058552 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 10631
I20260924 07:18:49.062151 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.053933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.3.246.126:46187)
0924 07:18:49.054000 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.054003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.054160 (+ 157us) server_negotiation.cc:407] Connection header received
0924 07:18:49.054577 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.054577 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.054588 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.054660 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.055241 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.055759 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.056696 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.056880 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.056936 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.059137 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.059139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.059140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.059148 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.060719 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.061170 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.061171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.061172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.061231 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.061476 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.061477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.061477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.061612 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.061664 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.061977 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.062058 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:18:49.062458 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.053870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:60664)
0924 07:18:49.054028 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.054069 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.054494 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.054726 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.054729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.054969 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.055154 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.055159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.055845 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.055846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.056606 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.056607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.056956 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.056956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.057056 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.057496 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.057503 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.059024 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.060798 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.060803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.060805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.061091 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.061294 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.061295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.061295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.061415 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.061710 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.061715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.061851 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.061934 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.062046 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260924 07:18:49.063169 10699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.246.125@KRBTEST.COM
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.125@KRBTEST.COM for oryx/127.3.246.125@KRBTEST.COM
I20260924 07:18:49.068652 10702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.059058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35202 (local address 127.3.246.125:35077)
0924 07:18:49.059446 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.059450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.059680 (+ 230us) server_negotiation.cc:407] Connection header received
0924 07:18:49.059723 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.059725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.059737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.059799 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.060590 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.061614 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.062625 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.063060 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.063127 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.065918 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.065924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.065925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.065930 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.067484 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.067839 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.067842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.067843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.067911 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.068109 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.068110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.068110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.068247 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.068420 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.068474 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.068548 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":78,"threads_started":1}
I20260924 07:18:49.070981 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.063420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35077 (local address 127.0.0.1:35210)
0924 07:18:49.063471 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.063480 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.063515 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.063722 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.063723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.063838 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.063882 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.063883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.064816 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.064817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.065587 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.065589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.066077 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.066078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.066158 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.066472 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.066478 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.067767 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.070026 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.070027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.070027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.070186 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.070399 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.070400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.070400 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.070454 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.070687 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.070688 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.070723 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.070789 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.070880 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 07:18:49.071075 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.063465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.3.246.125:35077)
0924 07:18:49.063594 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.063595 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.063602 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:18:49.063620 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.063621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.063630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.063693 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.063961 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.064738 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.065661 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.066011 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.066059 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.067851 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.067852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.067852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.067858 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.069933 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.070267 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.070268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.070269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.070326 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.070521 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.070522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.070522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.070622 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.070695 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.070824 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.070929 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260924 07:18:49.072193 10699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: Bootstrap starting.
I20260924 07:18:49.072841 10699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:49.073117 10699 log.cc:824] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:49.073889 10699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: No bootstrap required, opened a new log
I20260924 07:18:49.075523 10699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } }
I20260924 07:18:49.075639 10699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:49.075660 10699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13af89bf44fa42fe9c5a301d79df7ac2, State: Initialized, Role: FOLLOWER
I20260924 07:18:49.075804 10699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } }
I20260924 07:18:49.076143 10699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:49.076105 10706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } } }
I20260924 07:18:49.076244 10706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:18:49.077633 10717 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:18:49.077721 10717 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:18:49.079345 10631 master_runner.cc:428] Detected that this master 13af89bf44fa42fe9c5a301d79df7ac2 is joining an existing cluster
I20260924 07:18:49.079385 10631 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.246.125:35077
I20260924 07:18:49.087455 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.080538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.3.246.125:35077)
0924 07:18:49.080571 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.080573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.080614 (+ 41us) server_negotiation.cc:407] Connection header received
0924 07:18:49.080670 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.080671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.080680 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.080731 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.081229 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.082003 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.083407 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.083721 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.083804 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.084804 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.084805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.084806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.084812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.086216 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.086710 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.086711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.086711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.086754 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.086950 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.086950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.086951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.087064 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.087106 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.087258 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.087353 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":17}
I20260924 07:18:49.087690 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.080441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35077 (local address 127.0.0.1:35214)
0924 07:18:49.080565 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.080574 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.080623 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.080741 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.080742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.081034 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.081145 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.081147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.082084 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.082085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.082874 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.082876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.083810 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.083811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.083879 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.084256 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.084261 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.084682 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.086325 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.086325 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.086326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.086632 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.086817 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.086817 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.086817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.086884 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.087136 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.087136 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.087160 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.087237 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.087422 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":100,"thread_start_us":62,"threads_started":1}
I20260924 07:18:49.087898 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.080321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.3.246.126:46187)
0924 07:18:49.080365 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.080367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.080513 (+ 146us) server_negotiation.cc:407] Connection header received
0924 07:18:49.080612 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.080613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.080625 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.080657 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.081246 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.081815 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.082965 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.083276 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.083351 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.084527 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.084529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.084530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.084538 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.086204 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.086715 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.086716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.086717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.086806 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.087026 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.087027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.087027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.087156 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.087227 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.087714 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.087798 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:18:49.088205 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.080237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:60676)
0924 07:18:49.080452 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.080464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.080528 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.080724 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.080725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.081036 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.081145 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.081147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.081909 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.081910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.082874 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.082876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.083353 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.083354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.083403 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.084039 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.084043 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.084434 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.086302 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.086303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.086303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.086632 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.086879 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.086880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.086880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.086957 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.087544 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.087545 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.087584 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.088004 (+ 420us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.088104 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":15,"thread_start_us":85,"threads_started":1}
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234328, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.126@KRBTEST.COM for oryx/127.3.246.125@KRBTEST.COM
I20260924 07:18:49.097322 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.090214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.3.246.125:35077)
0924 07:18:49.090253 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.090255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.090343 (+ 88us) server_negotiation.cc:407] Connection header received
0924 07:18:49.090439 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.090439 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.090447 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.090491 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.090970 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.091380 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.092478 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.092644 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.092693 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.094775 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.094785 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.094785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.094792 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.096066 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.096525 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.096526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.096526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.096585 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.096793 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.096794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.096795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.096897 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.096970 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.097160 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.097234 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:18:49.097347 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.090101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35077 (local address 127.0.0.1:35216)
0924 07:18:49.090243 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.090292 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.090365 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.090497 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.090499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.090828 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.090897 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.090900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.091455 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.091457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.092407 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.092409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.092714 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.092715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.092803 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.093191 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.093196 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.094682 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.096153 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.096161 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.096163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.096431 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.096664 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.096665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.096665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.096729 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.096976 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.096984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.097032 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.097122 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.097238 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":89,"threads_started":1}
I20260924 07:18:49.097963 10167 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.246.125:35077: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2"
member_type: NON_VOTER
last_known_addr {
host: "127.3.246.125"
port: 35077
}
attrs {
promote: true
}
}
dest_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab"
cas_config_opid_index: -1
I20260924 07:18:49.098284 10167 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } }
I20260924 07:18:49.098811 10727 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 13af89bf44fa42fe9c5a301d79df7ac2 (127.3.246.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } } }
I20260924 07:18:49.099151 10673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:49.099893 10673 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 FOLLOWER]: Refusing update from remote peer e4f9ee1f206c43caa1a7f55aae00a8ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 07:18:49.100242 10726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:49.100934 10727 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.246.125:35077
I20260924 07:18:49.101018 10727 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } } }
I20260924 07:18:49.101090 10727 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:49.101167 10631 master.cc:537] Master@127.3.246.125:35077 shutting down...
I20260924 07:18:49.102098 10673 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 13af89bf44fa42fe9c5a301d79df7ac2 (127.3.246.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } } }
I20260924 07:18:49.103266 10729 mvcc.cc:205] Tried to move back new op lower bound from 7332799809266352128 to 7332799809139429376. Current Snapshot: MvccSnapshot[applied={T|T < 7332799809150156800}]
I20260924 07:18:49.103516 10706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4f9ee1f206c43caa1a7f55aae00a8ab. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } } }
I20260924 07:18:49.103644 10706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:18:49.103691 10706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } } }
I20260924 07:18:49.103727 10706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:18:49.103758 10706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } } }
I20260924 07:18:49.103807 10706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:18:49.104966 10631 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 LEARNER]: Raft consensus shutting down.
I20260924 07:18:49.105093 10631 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 LEARNER]: Raft consensus is shut down!
I20260924 07:18:49.105134 10631 tablet_replica.cc:330] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: stopping tablet replica
I20260924 07:18:49.120921 10631 master.cc:552] Master@127.3.246.125:35077 shutdown complete.
I20260924 07:18:49.120966 10631 master_runner.cc:305] Clearing existing system tablet
I20260924 07:18:49.121111 10631 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:49.121156 10631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:49.121855 10631 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:49.122385 10737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.122524 10631 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:49.122581 10631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.122603 10631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/wal
uuid: "13af89bf44fa42fe9c5a301d79df7ac2"
format_stamp: "Formatted at 2026-09-24 07:18:49 on dist-test-slave-rc1w"
server_key: "73b90505be868e932bc0319579f5d880"
server_key_iv: "15be502236ae51af34c11dd2a75b26ff"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.123188 10631 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:49.124684 10631 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 07:18:49.124739 10631 log.cc:1196] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 07:18:49.124964 10631 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: Deleting consensus metadata
I20260924 07:18:49.125084 10631 master_runner.cc:315] Copying system tablet from 127.3.246.126:46187
I20260924 07:18:49.125712 10631 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: tablet copy: Beginning tablet copy session from remote peer at address 127.3.246.126:46187
I20260924 07:18:49.131695 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.126071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60682 (local address 127.3.246.126:46187)
0924 07:18:49.126124 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.126125 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.126175 (+ 50us) server_negotiation.cc:407] Connection header received
0924 07:18:49.126259 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.126260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.126271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.126327 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.126670 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.127169 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.127903 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.128101 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.128177 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.129192 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.129193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.129194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.129203 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.130566 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.130921 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.130922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.130923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.130982 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.131183 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.131183 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.131184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.131299 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.131378 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.131528 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.131603 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:18:49.131711 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.125989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:60682)
0924 07:18:49.126102 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.126115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.126194 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.126348 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.126350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.126527 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.126583 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.126585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.127254 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.127255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.127817 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.127819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.128187 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.128189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.128258 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.128777 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.128782 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.129063 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.130666 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.130667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.130667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.130846 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.131051 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.131052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.131052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.131111 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.131369 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.131370 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.131415 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.131503 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.131604 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":68,"threads_started":1}
I20260924 07:18:49.132234 10187 tablet_copy_service.cc:140] P e4f9ee1f206c43caa1a7f55aae00a8ab: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 13af89bf44fa42fe9c5a301d79df7ac2 ({username='oryx', principal='oryx/127.3.246.125@KRBTEST.COM'} at 127.0.0.1:60682)
I20260924 07:18:49.132342 10187 tablet_copy_service.cc:161] P e4f9ee1f206c43caa1a7f55aae00a8ab: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 13af89bf44fa42fe9c5a301d79df7ac2 at {username='oryx', principal='oryx/127.3.246.125@KRBTEST.COM'} at 127.0.0.1:60682: session id = 13af89bf44fa42fe9c5a301d79df7ac2-00000000000000000000000000000000
I20260924 07:18:49.133188 10187 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 07:18:49.133829 10631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 07:18:49.134763 10631 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: tablet copy: Starting download of 0 data blocks...
I20260924 07:18:49.134932 10631 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: tablet copy: Starting download of 1 WAL segments...
I20260924 07:18:49.135756 10631 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 07:18:49.136521 10187 tablet_copy_service.cc:342] P e4f9ee1f206c43caa1a7f55aae00a8ab: Request end of tablet copy session 13af89bf44fa42fe9c5a301d79df7ac2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.246.125@KRBTEST.COM'} at 127.0.0.1:60682
I20260924 07:18:49.136580 10187 tablet_copy_service.cc:433] P e4f9ee1f206c43caa1a7f55aae00a8ab: ending tablet copy session 13af89bf44fa42fe9c5a301d79df7ac2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 13af89bf44fa42fe9c5a301d79df7ac2
I20260924 07:18:49.139007 10631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:49.140153 10745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.140215 10747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.140285 10744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:49.140581 10631 server_base.cc:1034] running on GCE node
I20260924 07:18:49.140679 10631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:49.140805 10631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:49.141925 10631 hybrid_clock.cc:648] HybridClock initialized: now 1790234329141912 us; error 34 us; skew 500 ppm
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:49.143132 10631 init.cc:377] Logged in from keytab as oryx/127.3.246.125@KRBTEST.COM (short username oryx)
I20260924 07:18:49.143424 10631 webserver.cc:460] Webserver started at http://127.3.246.125:36083/ using document root <none> and password file <none>
I20260924 07:18:49.143527 10631 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:49.143553 10631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:49.144166 10631 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:49.144665 10754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.144786 10631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:49.144833 10631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/wal
uuid: "13af89bf44fa42fe9c5a301d79df7ac2"
format_stamp: "Formatted at 2026-09-24 07:18:49 on dist-test-slave-rc1w"
server_key: "73b90505be868e932bc0319579f5d880"
server_key_iv: "15be502236ae51af34c11dd2a75b26ff"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.144891 10631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:49.164474 10631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:49.166096 10631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:49.169190 10631 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.125:35077
I20260924 07:18:49.169229 10806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.125:35077 every 8 connection(s)
I20260924 07:18:49.169518 10631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-1/data/info.pb
I20260924 07:18:49.170459 10807 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:18:49.170994 10807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: Bootstrap starting.
I20260924 07:18:49.173265 10807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:18:49.173484 10807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: Bootstrap complete.
I20260924 07:18:49.173717 10807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } }
I20260924 07:18:49.173817 10807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13af89bf44fa42fe9c5a301d79df7ac2, State: Initialized, Role: LEARNER
I20260924 07:18:49.173859 10807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } }
I20260924 07:18:49.174023 10811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } } }
I20260924 07:18:49.174026 10807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:49.174189 10811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:18:49.176378 10822 catalog_manager.cc:1259] Loaded cluster ID: f5a9a963570944cc95a964bb074d0315
I20260924 07:18:49.176426 10822 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: loading cluster ID for follower catalog manager: success
I20260924 07:18:49.177740 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.170134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.3.246.125:35077)
0924 07:18:49.170343 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.170345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.170351 (+ 6us) server_negotiation.cc:407] Connection header received
0924 07:18:49.170372 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.170372 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.170381 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.170456 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.171012 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.171658 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.172519 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.172730 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.172823 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.174329 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.174330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.174331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.174338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.176140 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.176662 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.176663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.176663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.176728 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.176946 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.176947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.176947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.177064 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.177152 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.177556 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.177627 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260924 07:18:49.178048 10822 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: acquiring CA information for follower catalog manager: success
I20260924 07:18:49.178369 10822 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for oryx/127.3.246.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.246.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.246.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.124.keytab.
Entry for principal oryx/127.3.246.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.124.keytab.
I20260924 07:18:49.204440 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.124:42711
--webserver_interface=127.3.246.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.124.keytab
--principal=oryx/127.3.246.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:46187,127.3.246.125:35077,127.3.246.124:42711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:49.285951 10825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:49.286141 10825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:49.286162 10825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:49.287297 10825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:49.287338 10825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:49.287352 10825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:49.287364 10825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:49.287376 10825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:49.288849 10825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:46187,127.3.246.125:35077,127.3.246.124:42711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.124.keytab
--principal=oryx/127.3.246.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.124:42711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.246.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:49.289078 10825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:49.289326 10825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:49.294148 10833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.294128 10831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.294142 10830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:49.294168 10825 server_base.cc:1034] running on GCE node
I20260924 07:18:49.294523 10825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:49.294847 10825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:49.296003 10825 hybrid_clock.cc:648] HybridClock initialized: now 1790234329295989 us; error 44 us; skew 500 ppm
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:49.298575 10825 init.cc:377] Logged in from keytab as oryx/127.3.246.124@KRBTEST.COM (short username oryx)
I20260924 07:18:49.299095 10825 webserver.cc:460] Webserver started at http://127.3.246.124:37937/ using document root <none> and password file <none>
I20260924 07:18:49.299340 10825 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:49.299405 10825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:49.299547 10825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:49.300549 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/data/instance:
uuid: "5738c38db05a44ea978f5e4fed86171f"
format_stamp: "Formatted at 2026-09-24 07:18:49 on dist-test-slave-rc1w"
server_key: "75c0aec818a486f9f6847d9a4763581c"
server_key_iv: "a71a9cbfc3df0526f6efe0cb29eb464f"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.300900 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/wal/instance:
uuid: "5738c38db05a44ea978f5e4fed86171f"
format_stamp: "Formatted at 2026-09-24 07:18:49 on dist-test-slave-rc1w"
server_key: "75c0aec818a486f9f6847d9a4763581c"
server_key_iv: "a71a9cbfc3df0526f6efe0cb29eb464f"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.302378 10825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:18:49.303100 10840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.303273 10825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:49.303359 10825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/wal
uuid: "5738c38db05a44ea978f5e4fed86171f"
format_stamp: "Formatted at 2026-09-24 07:18:49 on dist-test-slave-rc1w"
server_key: "75c0aec818a486f9f6847d9a4763581c"
server_key_iv: "a71a9cbfc3df0526f6efe0cb29eb464f"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.303407 10825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:49.333158 10825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:49.335218 10825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:49.335335 10825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:49.339363 10825 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.124:42711
I20260924 07:18:49.339407 10892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.124:42711 every 8 connection(s)
I20260924 07:18:49.339720 10825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/data/info.pb
I20260924 07:18:49.340621 10893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:49.342262 10893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.124@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:49.349553 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 10825
I20260924 07:18:49.351200 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.343118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.3.246.126:46187)
0924 07:18:49.343167 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.343171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.343207 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:18:49.343716 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.343718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.343730 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.343802 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.344396 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.344954 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.346056 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.346257 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.346321 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.348357 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.348359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.348359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.348366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.349594 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.350210 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.350212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.350212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.350277 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.350557 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.350558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.350559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.350710 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.350791 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.351031 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.351115 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:18:49.351333 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.342925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:60684)
0924 07:18:49.343112 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.343147 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.343582 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.343860 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.343863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.344098 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.344298 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.344302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.345030 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.345031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.345942 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.345946 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.346351 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.346352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.346451 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.346933 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.346941 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.348267 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.349867 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.349874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.349876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.350124 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.350354 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.350354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.350355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.350491 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.350783 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.350787 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.350893 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.350976 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.351087 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260924 07:18:49.351851 10893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.124@KRBTEST.COM for oryx/127.3.246.125@KRBTEST.COM
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.246.124@KRBTEST.COM
I20260924 07:18:49.359385 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.352088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35077 (local address 127.0.0.1:35240)
0924 07:18:49.352137 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.352148 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.352187 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.352309 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.352310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.352440 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.352489 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.352490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.353482 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.353483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.354208 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.354210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.354659 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.354660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.354734 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.355052 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.355058 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.356420 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.358242 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.358243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.358243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.358535 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.358734 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.358735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.358735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.358802 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.359034 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.359035 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.359084 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.359176 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.359269 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 07:18:49.359402 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.352139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.3.246.125:35077)
0924 07:18:49.352179 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.352181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.352189 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:18:49.352238 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.352239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.352250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.352303 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.352559 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.353318 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.354289 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.354576 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.354632 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.356520 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.356521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.356521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.356529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.358163 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.358615 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.358615 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.358616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.358665 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.358872 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.358872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.358872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.358967 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.359016 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.359222 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.359314 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:18:49.359876 10893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:18:49.360855 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.350137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.3.246.124:42711)
0924 07:18:49.350275 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.350279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.350300 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:18:49.350335 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.350337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.350351 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.350425 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.351147 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.352447 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.353461 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.353927 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.354013 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.357224 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.357231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.357232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.357238 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.359165 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.359753 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.359756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.359756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.359846 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.360214 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.360215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.360216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.360399 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.360558 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.360655 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.360726 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.124@KRBTEST.COM for oryx/127.3.246.124@KRBTEST.COM
I20260924 07:18:49.366508 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.360147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:42711 (local address 127.0.0.1:45620)
0924 07:18:49.360197 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.360207 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.360241 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.360528 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.360529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.360686 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.360720 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.360722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.361577 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.361578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.362097 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.362099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.362431 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.362432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.362469 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.362848 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.362853 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.363918 (+ 1065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.365753 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.365753 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.365754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.365902 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.366061 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.366061 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.366061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.366107 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.366299 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.366300 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.366332 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.366383 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.366452 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 07:18:49.366633 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.360236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45620 (local address 127.3.246.124:42711)
0924 07:18:49.360398 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.360400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.360405 (+ 5us) server_negotiation.cc:407] Connection header received
0924 07:18:49.360419 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.360419 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.360426 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.360505 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.360809 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.361472 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.362166 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.362367 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.362422 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.363994 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.363995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.363996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.364001 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.365605 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.365950 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.365951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.365951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.366003 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.366145 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.366145 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.366146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.366239 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.366289 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.366400 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.366498 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260924 07:18:49.367862 10893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: Bootstrap starting.
I20260924 07:18:49.368454 10893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:49.368712 10893 log.cc:824] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:49.369347 10893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: No bootstrap required, opened a new log
I20260924 07:18:49.371076 10893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } }
I20260924 07:18:49.371200 10893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:49.371225 10893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5738c38db05a44ea978f5e4fed86171f, State: Initialized, Role: FOLLOWER
I20260924 07:18:49.371343 10893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } }
I20260924 07:18:49.371595 10899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } } }
I20260924 07:18:49.371660 10893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:49.371707 10899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:18:49.373534 10910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:18:49.373625 10910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:18:49.375905 10825 master_runner.cc:428] Detected that this master 5738c38db05a44ea978f5e4fed86171f is joining an existing cluster
I20260924 07:18:49.375957 10825 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.246.124:42711
I20260924 07:18:49.384006 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.377005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.3.246.124:42711)
0924 07:18:49.377039 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.377042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.377323 (+ 281us) server_negotiation.cc:407] Connection header received
0924 07:18:49.377419 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.377419 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.377428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.377457 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.377954 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.378654 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.379702 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.379981 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.380024 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.381107 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.381108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.381109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.381114 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.382958 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.383357 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.383357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.383358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.383408 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.383595 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.383595 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.383595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.383678 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.383722 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.383890 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.383953 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 07:18:49.384374 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.377134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:42711 (local address 127.0.0.1:45626)
0924 07:18:49.377258 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.377268 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.377345 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.377517 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.377517 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.377800 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.377863 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.377865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.378757 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.378758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.379592 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.379595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.380122 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.380123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.380178 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.380576 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.380581 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.381005 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.383043 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.383044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.383044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.383282 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.383469 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.383470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.383470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.383535 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.383741 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.383742 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.383786 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.383878 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.384024 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":83,"thread_start_us":71,"threads_started":1}
I20260924 07:18:49.402114 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.377326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35077 (local address 127.0.0.1:35246)
0924 07:18:49.377449 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.377458 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.377541 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.377740 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.377741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.377927 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.377978 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.377980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.378847 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.378848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.379689 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.379691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.380201 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.380202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.380266 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.398541 (+ 18275us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.398545 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.399038 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.400917 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.400918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.400918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.401098 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.401301 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.401302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.401302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.401377 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.401722 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.401723 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.401778 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.401876 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.401988 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260924 07:18:49.402159 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.377234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.3.246.125:35077)
0924 07:18:49.377272 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.377274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.377504 (+ 230us) server_negotiation.cc:407] Connection header received
0924 07:18:49.377614 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.377615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.377626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.377655 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.378061 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.378765 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.379788 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.380082 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.380124 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.399197 (+ 19073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.399198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.399217 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.399228 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.400798 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.401174 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.401176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.401176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.401236 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.401453 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.401454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.401454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.401591 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.401676 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.401931 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.402030 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:18:49.402491 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.377007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.3.246.126:46187)
0924 07:18:49.377057 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.377059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.377166 (+ 107us) server_negotiation.cc:407] Connection header received
0924 07:18:49.377275 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.377276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.377288 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.377321 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.377854 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.378621 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.379798 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.380089 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.380124 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.399180 (+ 19056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.399183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.399183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.399210 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.401148 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.401507 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.401508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.401508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.401567 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.401844 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.401845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.401846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.401987 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.402033 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.402284 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.402379 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:18:49.402832 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.376930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:60694)
0924 07:18:49.377105 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.377115 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.377196 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.377392 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.377393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.377655 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.377753 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.377754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.378708 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.378708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.379696 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.379697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.380219 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.380219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.380269 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.398455 (+ 18186us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.398460 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.399056 (+ 596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.401240 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.401241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.401241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.401439 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.401637 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.401638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.401638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.401742 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.402082 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.402082 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.402118 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.402572 (+ 454us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.402712 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":76,"thread_start_us":132,"threads_started":1}
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234328, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.126@KRBTEST.COM for oryx/127.3.246.124@KRBTEST.COM
I20260924 07:18:49.411556 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.404843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:42711 (local address 127.0.0.1:45638)
0924 07:18:49.404897 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.404910 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.404950 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.405125 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.405126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.405275 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.405313 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.405315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.406082 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.406083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.406592 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.406593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.407000 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.407001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.407082 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.407503 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.407508 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.408895 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.410617 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.410618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.410619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.410779 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.411002 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.411003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.411003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.411064 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.411333 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.411334 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.411367 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.411418 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.411492 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 07:18:49.411720 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.404921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.3.246.124:42711)
0924 07:18:49.405020 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.405021 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.405027 (+ 6us) server_negotiation.cc:407] Connection header received
0924 07:18:49.405050 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.405051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.405058 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.405121 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.405397 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.405977 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.406731 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.406923 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.407002 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.408987 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.408989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.408989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.408996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.410527 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.410869 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.410870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.410870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.410932 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.411133 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.411134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.411134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.411265 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.411339 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.411459 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.411537 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 07:18:49.412245 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 10229
I20260924 07:18:49.412317 10167 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.246.124:42711: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5738c38db05a44ea978f5e4fed86171f"
member_type: NON_VOTER
last_known_addr {
host: "127.3.246.124"
port: 42711
}
attrs {
promote: true
}
}
dest_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab"
cas_config_opid_index: 5
I20260924 07:18:49.412554 10167 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } }
I20260924 07:18:49.413185 10726 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5738c38db05a44ea978f5e4fed86171f (127.3.246.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } }
I20260924 07:18:49.414410 10726 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.246.124:42711
I20260924 07:18:49.414484 10726 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } }
I20260924 07:18:49.414568 10726 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:49.414885 10825 master.cc:537] Master@127.3.246.124:42711 shutting down...
I20260924 07:18:49.415068 10867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:49.416178 10867 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 1 FOLLOWER]: Refusing update from remote peer e4f9ee1f206c43caa1a7f55aae00a8ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 07:18:49.416463 10919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:18:49.417027 10154 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 07:18:49.417512 10154 consensus_peers.cc:487] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab -> Peer 5738c38db05a44ea978f5e4fed86171f (127.3.246.124:42711): Couldn't send request to peer 5738c38db05a44ea978f5e4fed86171f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 07:18:49.420724 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:41413
--local_ip_for_outbound_sockets=127.3.246.65
--tserver_master_addrs=127.3.246.126:46187
--webserver_port=42913
--webserver_interface=127.3.246.65
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab
--principal=oryx/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.246.126:46187,127.3.246.125:35077
--tserver_master_addrs=127.3.246.126:46187,127.3.246.125:35077,127.3.246.124:42711 with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:18:49.422448 10825 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:49.422552 10825 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:49.422609 10825 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: stopping tablet replica
I20260924 07:18:49.425563 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.418011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.3.246.125:35077)
0924 07:18:49.418085 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.418087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.420008 (+ 1921us) server_negotiation.cc:407] Connection header received
0924 07:18:49.420064 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.420065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.420092 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.420171 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:49.421172 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.421944 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.424208 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.425140 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.425227 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.425295 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.425451 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.425491 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:18:49.425585 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.413332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35077 (local address 127.0.0.1:35254)
0924 07:18:49.419924 (+ 6592us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.419938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.419995 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.421012 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.421013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.421015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:49.421060 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.421061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.422072 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.422073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.424094 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.424097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.425240 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.425242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.425337 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.425363 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.425436 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.425485 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6573}
I20260924 07:18:49.425868 10781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 LEARNER]: Refusing update from remote peer e4f9ee1f206c43caa1a7f55aae00a8ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 07:18:49.426124 10919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 07:18:49.426584 10781 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5738c38db05a44ea978f5e4fed86171f (127.3.246.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } }
I20260924 07:18:49.427865 10811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4f9ee1f206c43caa1a7f55aae00a8ab. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } }
I20260924 07:18:49.427978 10811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:18:49.428022 10811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: NON_VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: true } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } }
I20260924 07:18:49.428089 10811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:18:49.428349 10919 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab: attempting to promote NON_VOTER 13af89bf44fa42fe9c5a301d79df7ac2 to VOTER
I20260924 07:18:49.428584 10919 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } }
I20260924 07:18:49.429145 10781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 LEARNER]: Refusing update from remote peer e4f9ee1f206c43caa1a7f55aae00a8ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 07:18:49.429345 10726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 07:18:49.429842 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.428975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:42711 (local address 127.0.0.1:45642)
0924 07:18:49.429756 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.429805 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":762}
I20260924 07:18:49.430187 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.430065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:42711 (local address 127.0.0.1:45658)
0924 07:18:49.430101 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.430124 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 07:18:49.430104 10726 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 13af89bf44fa42fe9c5a301d79df7ac2 (127.3.246.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } }
I20260924 07:18:49.430506 10781 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 13af89bf44fa42fe9c5a301d79df7ac2 (127.3.246.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } }
I20260924 07:18:49.431125 10726 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } }
I20260924 07:18:49.431212 10726 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:49.431270 10726 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } }
I20260924 07:18:49.431339 10726 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: This master's current role is: LEADER
W20260924 07:18:49.431492 10154 consensus_peers.cc:487] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab -> Peer 5738c38db05a44ea978f5e4fed86171f (127.3.246.124:42711): Couldn't send request to peer 5738c38db05a44ea978f5e4fed86171f. Status: Network error: Client connection negotiation failed: client connection to 127.3.246.124:42711: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 07:18:49.431480 10811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } }
I20260924 07:18:49.431559 10811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:18:49.436206 10825 master.cc:552] Master@127.3.246.124:42711 shutdown complete.
I20260924 07:18:49.436260 10825 master_runner.cc:305] Clearing existing system tablet
I20260924 07:18:49.436403 10825 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:49.436452 10825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:49.437301 10825 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:18:49.437896 10925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.438047 10825 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:49.438122 10825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.438151 10825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/wal
uuid: "5738c38db05a44ea978f5e4fed86171f"
format_stamp: "Formatted at 2026-09-24 07:18:49 on dist-test-slave-rc1w"
server_key: "75c0aec818a486f9f6847d9a4763581c"
server_key_iv: "a71a9cbfc3df0526f6efe0cb29eb464f"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.438947 10825 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:49.440457 10825 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 07:18:49.440533 10825 log.cc:1196] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 07:18:49.440748 10825 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: Deleting consensus metadata
I20260924 07:18:49.441043 10825 master_runner.cc:315] Copying system tablet from 127.3.246.126:46187
I20260924 07:18:49.441741 10825 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: tablet copy: Beginning tablet copy session from remote peer at address 127.3.246.126:46187
I20260924 07:18:49.451946 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.442105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.3.246.126:46187)
0924 07:18:49.442161 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.442163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.445003 (+ 2840us) server_negotiation.cc:407] Connection header received
0924 07:18:49.445070 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.445071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.445087 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.445173 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.445532 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.446402 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.447410 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.447685 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.447767 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.448956 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.448958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.448958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.448967 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.450674 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.451113 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.451114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.451114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.451177 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.451401 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.451402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.451402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.451531 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.451599 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.451785 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.451859 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:18:49.451957 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.442026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:60706)
0924 07:18:49.444898 (+ 2872us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.444912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.445011 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.445186 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.445187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.445392 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.445439 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.445440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.446506 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.446508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.447311 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.447314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.447833 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.447834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.447903 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.448441 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.448447 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.448853 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.450773 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.450774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.450775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.451019 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.451251 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.451252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.451253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.451327 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.451608 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.451609 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.451662 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.451748 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.451860 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2849,"thread_start_us":64,"threads_started":1}
I20260924 07:18:49.452560 10187 tablet_copy_service.cc:140] P e4f9ee1f206c43caa1a7f55aae00a8ab: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5738c38db05a44ea978f5e4fed86171f ({username='oryx', principal='oryx/127.3.246.124@KRBTEST.COM'} at 127.0.0.1:60706)
I20260924 07:18:49.452644 10187 tablet_copy_service.cc:161] P e4f9ee1f206c43caa1a7f55aae00a8ab: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5738c38db05a44ea978f5e4fed86171f at {username='oryx', principal='oryx/127.3.246.124@KRBTEST.COM'} at 127.0.0.1:60706: session id = 5738c38db05a44ea978f5e4fed86171f-00000000000000000000000000000000
I20260924 07:18:49.453061 10187 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 07:18:49.453567 10825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 07:18:49.454960 10825 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: tablet copy: Starting download of 0 data blocks...
I20260924 07:18:49.455152 10825 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: tablet copy: Starting download of 1 WAL segments...
I20260924 07:18:49.456032 10825 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 07:18:49.456967 10187 tablet_copy_service.cc:342] P e4f9ee1f206c43caa1a7f55aae00a8ab: Request end of tablet copy session 5738c38db05a44ea978f5e4fed86171f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.246.124@KRBTEST.COM'} at 127.0.0.1:60706
I20260924 07:18:49.457027 10187 tablet_copy_service.cc:433] P e4f9ee1f206c43caa1a7f55aae00a8ab: ending tablet copy session 5738c38db05a44ea978f5e4fed86171f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5738c38db05a44ea978f5e4fed86171f
I20260924 07:18:49.460048 10825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:49.461256 10932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.462918 10935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:49.463058 10825 server_base.cc:1034] running on GCE node
W20260924 07:18:49.468395 10933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:49.468601 10825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:49.468778 10825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:49.469920 10825 hybrid_clock.cc:648] HybridClock initialized: now 1790234329469899 us; error 40 us; skew 500 ppm
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:49.471524 10825 init.cc:377] Logged in from keytab as oryx/127.3.246.124@KRBTEST.COM (short username oryx)
I20260924 07:18:49.471788 10825 webserver.cc:460] Webserver started at http://127.3.246.124:35359/ using document root <none> and password file <none>
I20260924 07:18:49.471905 10825 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:49.471939 10825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:49.472702 10825 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:49.473337 10942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.473464 10825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.473549 10825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/wal
uuid: "5738c38db05a44ea978f5e4fed86171f"
format_stamp: "Formatted at 2026-09-24 07:18:49 on dist-test-slave-rc1w"
server_key: "75c0aec818a486f9f6847d9a4763581c"
server_key_iv: "a71a9cbfc3df0526f6efe0cb29eb464f"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.473627 10825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:49.480752 10825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:49.482614 10825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:49.485980 10825 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.124:42711
I20260924 07:18:49.486340 10825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/master-2/data/info.pb
I20260924 07:18:49.487145 10995 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:18:49.487690 10995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: Bootstrap starting.
I20260924 07:18:49.489794 10994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.124:42711 every 8 connection(s)
I20260924 07:18:49.490509 10995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:18:49.490734 10995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: Bootstrap complete.
I20260924 07:18:49.490922 10995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } }
I20260924 07:18:49.491012 10995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5738c38db05a44ea978f5e4fed86171f, State: Initialized, Role: LEARNER
I20260924 07:18:49.491045 10995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } }
I20260924 07:18:49.491195 10995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:49.492528 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } }
I20260924 07:18:49.492633 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: This master's current role is: LEARNER
I20260924 07:18:49.492977 11009 catalog_manager.cc:1259] Loaded cluster ID: f5a9a963570944cc95a964bb074d0315
I20260924 07:18:49.493063 11009 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: loading cluster ID for follower catalog manager: success
I20260924 07:18:49.494241 11009 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: acquiring CA information for follower catalog manager: success
I20260924 07:18:49.494542 11009 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 07:18:49.519124 10921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:49.519343 10921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:49.519379 10921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:49.520514 10921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:49.520579 10921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:49.520624 10921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:49.522156 10921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab
--principal=oryx/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:41413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=42913
--tserver_master_addrs=127.3.246.126:46187,127.3.246.125:35077,127.3.246.124:42711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:49.522415 10921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:49.522657 10921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:49.527648 11015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.527686 11014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.527675 11017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:49.528080 10921 server_base.cc:1034] running on GCE node
I20260924 07:18:49.528276 10921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:49.528537 10921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:49.529711 10921 hybrid_clock.cc:648] HybridClock initialized: now 1790234329529695 us; error 38 us; skew 500 ppm
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:49.532147 10921 init.cc:377] Logged in from keytab as oryx/127.3.246.65@KRBTEST.COM (short username oryx)
I20260924 07:18:49.532549 10921 webserver.cc:460] Webserver started at http://127.3.246.65:42913/ using document root <none> and password file <none>
I20260924 07:18:49.532820 10921 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:49.532907 10921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:49.534360 10921 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:49.534963 11024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.535131 10921 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:49.535230 10921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "7be36e4202744663900a72eb10a5ae86"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "2dde677999a5efd3f2c58fe105e38c5a"
server_key_iv: "4dbdb138342c293570ae5b8f0c2b0ec5"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.535578 10921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:49.552644 10921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:49.555053 10921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:49.555220 10921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:49.555472 10921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:49.555768 10921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:49.555823 10921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.555867 10921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:49.555893 10921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.562000 10921 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:41413
I20260924 07:18:49.562110 11137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:41413 every 8 connection(s)
I20260924 07:18:49.562420 10921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:49.566658 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 10921
I20260924 07:18:49.566838 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 10363
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.124@KRBTEST.COM
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.125@KRBTEST.COM
I20260924 07:18:49.574626 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:34417
--local_ip_for_outbound_sockets=127.3.246.66
--tserver_master_addrs=127.3.246.126:46187
--webserver_port=45225
--webserver_interface=127.3.246.66
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab
--principal=oryx/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.246.126:46187,127.3.246.125:35077
--tserver_master_addrs=127.3.246.126:46187,127.3.246.125:35077,127.3.246.124:42711 with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:18:49.575924 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.563153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:54117 (local address 127.3.246.125:35077)
0924 07:18:49.563217 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.563220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.563487 (+ 267us) server_negotiation.cc:407] Connection header received
0924 07:18:49.564570 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.564571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.564586 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.564633 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.565356 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.566090 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.567777 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.568078 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.568123 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.571926 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.571927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.571927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.571935 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.573811 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.574670 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.574672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.574672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.574734 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.575013 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.575013 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.575014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.575175 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.575222 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.575725 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.575829 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:18:49.575937 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.563591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:35263 (local address 127.3.246.126:46187)
0924 07:18:49.563716 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.563718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.563728 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:18:49.564569 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.564571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.564586 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.564624 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.565753 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.566245 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.567783 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.568102 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.568159 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.571966 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.571968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.571968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.571976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.574049 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.574627 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.574629 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.574629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.574699 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.575013 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.575014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.575014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.575148 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.575202 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.575779 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.575854 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 07:18:49.576543 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.563478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.65:35263)
0924 07:18:49.563608 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.563620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.564435 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.564725 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.564726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.565081 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.565214 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.565217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.566356 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.566357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.567672 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.567675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.568244 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.568245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.568368 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.569050 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.569060 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.571706 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.574344 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.574345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.574346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.574560 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.574785 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.574786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.574786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.574925 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.575259 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.575265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.575450 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.576071 (+ 621us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.576304 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":11,"thread_start_us":61,"threads_started":1}
I20260924 07:18:49.576546 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.563169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35077 (local address 127.3.246.65:54117)
0924 07:18:49.563352 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.563388 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.564440 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.564725 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.564726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.565086 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.565216 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.565217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.566209 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.566210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.567669 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.567672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.568216 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.568217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.568368 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.569091 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.569096 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.571856 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.573910 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.573917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.573919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.574262 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.574818 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.574818 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.574819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.574925 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.575274 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.575274 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.575442 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.576071 (+ 629us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.576352 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":295,"thread_start_us":99,"threads_started":1}
I20260924 07:18:49.576776 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.563162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:36553 (local address 127.3.246.124:42711)
0924 07:18:49.563294 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.563296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.563494 (+ 198us) server_negotiation.cc:407] Connection header received
0924 07:18:49.564506 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.564508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.564522 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.564592 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.565258 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.566299 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.567776 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.568090 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.568127 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.571868 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.571869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.571869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.571877 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.574622 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.575376 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.575378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.575378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.575438 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.575641 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.575642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.575642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.575778 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.575942 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.576555 (+ 613us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.576645 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260924 07:18:49.577036 11140 heartbeater.cc:347] Connected to a master server at 127.3.246.125:35077
I20260924 07:18:49.577134 11140 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:49.577275 11138 heartbeater.cc:347] Connected to a master server at 127.3.246.126:46187
I20260924 07:18:49.577332 11140 heartbeater.cc:511] Master 127.3.246.125:35077 requested a full tablet report, sending...
I20260924 07:18:49.577345 11138 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:49.577454 11138 heartbeater.cc:511] Master 127.3.246.126:46187 requested a full tablet report, sending...
I20260924 07:18:49.577564 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.563169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:42711 (local address 127.3.246.65:36553)
0924 07:18:49.563344 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.563387 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.564396 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.564658 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.564662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.564960 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.565154 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.565158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.566385 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.566385 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.567669 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.567672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.568217 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.568218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.568368 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.569226 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.569231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.571766 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.575053 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.575054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.575055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.575279 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.575516 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.575517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.575517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.575577 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.576396 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.576397 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.576443 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.576527 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.577247 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":87,"thread_start_us":90,"threads_started":1}
I20260924 07:18:49.577826 10167 ts_manager.cc:194] Re-registered known tserver with Master: 7be36e4202744663900a72eb10a5ae86 (127.3.246.65:41413)
I20260924 07:18:49.577960 11139 heartbeater.cc:347] Connected to a master server at 127.3.246.124:42711
I20260924 07:18:49.578008 11139 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:49.578076 11139 heartbeater.cc:511] Master 127.3.246.124:42711 requested a full tablet report, sending...
I20260924 07:18:49.578310 10167 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:35263
I20260924 07:18:49.578437 10959 ts_manager.cc:194] Registered new tserver with Master: 7be36e4202744663900a72eb10a5ae86 (127.3.246.65:41413)
I20260924 07:18:49.578614 10771 ts_manager.cc:194] Registered new tserver with Master: 7be36e4202744663900a72eb10a5ae86 (127.3.246.65:41413)
W20260924 07:18:49.658321 11146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:49.658556 11146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:49.658597 11146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:49.659775 11146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:49.659844 11146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:49.659889 11146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:49.661456 11146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab
--principal=oryx/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:34417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=45225
--tserver_master_addrs=127.3.246.126:46187,127.3.246.125:35077,127.3.246.124:42711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:49.661753 11146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:49.661988 11146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:49.667107 11152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.667092 11154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.667164 11151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:49.667834 11146 server_base.cc:1034] running on GCE node
I20260924 07:18:49.668080 11146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:49.668352 11146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:49.669502 11146 hybrid_clock.cc:648] HybridClock initialized: now 1790234329669493 us; error 43 us; skew 500 ppm
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:49.672082 11146 init.cc:377] Logged in from keytab as oryx/127.3.246.66@KRBTEST.COM (short username oryx)
I20260924 07:18:49.672590 11146 webserver.cc:460] Webserver started at http://127.3.246.66:45225/ using document root <none> and password file <none>
I20260924 07:18:49.672825 11146 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:49.672909 11146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:49.674685 11146 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:49.675352 11161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.675587 11146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:49.675686 11146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "878909796545ed548a24feda4177b9fb"
server_key_iv: "02f4703c77f0b9c1afd4c3e72fc40fdc"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.676098 11146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:49.687945 11146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:49.690276 11146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:49.690449 11146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:49.690687 11146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:49.691040 11146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:49.691097 11146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.691131 11146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:49.691159 11146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.697093 11146 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:34417
I20260924 07:18:49.697134 11274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:34417 every 8 connection(s)
I20260924 07:18:49.697480 11146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:49.701527 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 11146
I20260924 07:18:49.701748 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 10497
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.124@KRBTEST.COM
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.125@KRBTEST.COM
I20260924 07:18:49.708526 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:33321
--local_ip_for_outbound_sockets=127.3.246.67
--tserver_master_addrs=127.3.246.126:46187
--webserver_port=40789
--webserver_interface=127.3.246.67
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab
--principal=oryx/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.246.126:46187,127.3.246.125:35077
--tserver_master_addrs=127.3.246.126:46187,127.3.246.125:35077,127.3.246.124:42711 with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:18:49.710124 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.698612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:44821 (local address 127.3.246.125:35077)
0924 07:18:49.698655 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.698657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.698872 (+ 215us) server_negotiation.cc:407] Connection header received
0924 07:18:49.699305 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.699307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.699321 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.699354 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.700317 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.701083 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.704213 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.704530 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.704573 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.706581 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.706582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.706582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.706590 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.708337 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.708994 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.708995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.708996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.709066 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.709372 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.709373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.709374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.709498 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.709542 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.709947 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.710029 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:18:49.710125 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.698294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:48465 (local address 127.3.246.124:42711)
0924 07:18:49.698350 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.698352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.698741 (+ 389us) server_negotiation.cc:407] Connection header received
0924 07:18:49.699305 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.699307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.699323 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.699354 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.700318 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.701084 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.702279 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.702585 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.702626 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.706580 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.706582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.706582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.706590 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.708514 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.708994 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.708995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.708995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.709066 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.709364 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.709365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.709365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.709498 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.709542 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.709947 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.710031 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 07:18:49.710212 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.698309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:57123 (local address 127.3.246.126:46187)
0924 07:18:49.698680 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.698682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.698694 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:18:49.699228 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.699230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.699243 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.699313 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.700330 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.700822 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.702264 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.702592 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.702635 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.706266 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.706268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.706268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.706276 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.708899 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.709311 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.709312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.709313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.709368 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.709558 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.709559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.709559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.709709 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.709759 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.709996 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.710087 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334}
I20260924 07:18:49.710733 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.698520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35077 (local address 127.3.246.66:44821)
0924 07:18:49.698793 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.698804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.699181 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.699426 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.699426 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.699995 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.700205 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.700212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.701182 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.701183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.704099 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.704102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.704652 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.704653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.704729 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.705157 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.705163 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.706475 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.708500 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.708506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.708508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.708903 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.709136 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.709137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.709137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.709282 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.709593 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.709600 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.709784 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.710543 (+ 759us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.710638 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":51,"thread_start_us":71,"threads_started":1}
I20260924 07:18:49.710733 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.698249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:42711 (local address 127.3.246.66:48465)
0924 07:18:49.698645 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.698656 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.699183 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.699426 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.699426 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.699986 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.700205 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.700212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.701182 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.701183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.702159 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.702162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.702726 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.702727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.702861 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.703593 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.703598 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.706475 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.708591 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.708592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.708592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.708903 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.709140 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.709141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.709141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.709282 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.709593 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.709600 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.709784 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.710254 (+ 470us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.710444 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":36,"thread_start_us":98,"threads_started":1}
I20260924 07:18:49.710928 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.698248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.66:57123)
0924 07:18:49.698417 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.698459 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.699114 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.699384 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.699388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.700002 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.700204 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.700209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.700934 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.700935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.702162 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.702165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.702719 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.702720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.702868 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.703531 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.703541 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.706156 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.708997 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.708998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.708999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.709230 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.709436 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.709437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.709437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.709495 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.709824 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.709825 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.709871 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.710335 (+ 464us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.710515 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":75,"thread_start_us":96,"threads_started":1}
I20260924 07:18:49.711490 11277 heartbeater.cc:347] Connected to a master server at 127.3.246.125:35077
I20260924 07:18:49.711488 11275 heartbeater.cc:347] Connected to a master server at 127.3.246.126:46187
I20260924 07:18:49.711488 11276 heartbeater.cc:347] Connected to a master server at 127.3.246.124:42711
I20260924 07:18:49.711609 11275 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:49.711609 11276 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:49.711611 11277 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:49.711838 11277 heartbeater.cc:511] Master 127.3.246.125:35077 requested a full tablet report, sending...
I20260924 07:18:49.711836 11276 heartbeater.cc:511] Master 127.3.246.124:42711 requested a full tablet report, sending...
I20260924 07:18:49.711836 11275 heartbeater.cc:511] Master 127.3.246.126:46187 requested a full tablet report, sending...
I20260924 07:18:49.712265 10167 ts_manager.cc:194] Re-registered known tserver with Master: bf13e4ca3f8c4b08bedf0e3654c3e86e (127.3.246.66:34417)
I20260924 07:18:49.712321 10959 ts_manager.cc:194] Registered new tserver with Master: bf13e4ca3f8c4b08bedf0e3654c3e86e (127.3.246.66:34417)
I20260924 07:18:49.712347 10771 ts_manager.cc:194] Registered new tserver with Master: bf13e4ca3f8c4b08bedf0e3654c3e86e (127.3.246.66:34417)
I20260924 07:18:49.712754 10167 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:57123
W20260924 07:18:49.793732 11282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:49.794049 11282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:49.794085 11282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:49.795425 11282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:49.795481 11282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:49.795523 11282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:49.797075 11282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab
--principal=oryx/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:33321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=40789
--tserver_master_addrs=127.3.246.126:46187,127.3.246.125:35077,127.3.246.124:42711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:49.797253 11282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:49.797425 11282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:49.802471 11288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.802507 11287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.802482 11290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:49.802839 11282 server_base.cc:1034] running on GCE node
I20260924 07:18:49.803052 11282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:49.803347 11282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:49.804557 11282 hybrid_clock.cc:648] HybridClock initialized: now 1790234329804530 us; error 42 us; skew 500 ppm
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:49.807060 11282 init.cc:377] Logged in from keytab as oryx/127.3.246.67@KRBTEST.COM (short username oryx)
I20260924 07:18:49.807416 11282 webserver.cc:460] Webserver started at http://127.3.246.67:40789/ using document root <none> and password file <none>
I20260924 07:18:49.807588 11282 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:49.807639 11282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:49.808905 11282 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:49.809487 11297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.809640 11282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:18:49.809753 11282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "16eef29dfa904346bcb8ff1485075d58"
format_stamp: "Formatted at 2026-09-24 07:18:48 on dist-test-slave-rc1w"
server_key: "f80b8fe5829636208db4627fe96e173e"
server_key_iv: "535b657beaafd19844eb1512b9e0de74"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.810063 11282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:49.825169 11282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:49.827478 11282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:49.827589 11282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:49.827772 11282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:49.828022 11282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:49.828049 11282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.828078 11282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:49.828091 11282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.833845 11282 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:33321
I20260924 07:18:49.833981 11410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:33321 every 8 connection(s)
I20260924 07:18:49.834246 11282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:49.836027 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 11282
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.124@KRBTEST.COM
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.125@KRBTEST.COM
Sep 24 07:18:49 dist-test-slave-rc1w krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:49.848320 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.835297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:41921 (local address 127.3.246.126:46187)
0924 07:18:49.835351 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.835353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.835511 (+ 158us) server_negotiation.cc:407] Connection header received
0924 07:18:49.836090 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.836091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.836104 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.836138 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.837288 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.838085 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.839615 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.839919 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.839959 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.844183 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.844184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.844185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.844192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.846700 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.847196 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.847197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.847198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.847257 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.847541 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.847542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.847543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.847685 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.847767 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.848136 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.848218 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:18:49.848986 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.835022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.67:41921)
0924 07:18:49.835438 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.835448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.835978 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.836328 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.836333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.836842 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.837168 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.837174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.838185 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.838185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.839514 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.839517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.840038 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.840039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.840149 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.840833 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.840844 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.844098 (+ 3254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.846795 (+ 2697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.846802 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.846804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.847101 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.847335 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.847336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.847336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.847460 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.847799 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.847805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.847967 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.848521 (+ 554us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.848711 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":86,"thread_start_us":69,"threads_started":1}
I20260924 07:18:49.849589 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.837239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45668 (local address 127.3.246.124:42711)
0924 07:18:49.838243 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.838245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.838555 (+ 310us) server_negotiation.cc:407] Connection header received
0924 07:18:49.838577 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.838578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.838589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.838643 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.839439 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.840150 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.841814 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.842141 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.842205 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.843865 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.843866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.843867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.843873 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.845806 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.846259 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.846260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.846260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.846318 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.846515 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.846516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.846516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.846664 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.846763 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.848065 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.848179 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987,"thread_start_us":73,"threads_started":1}
I20260924 07:18:49.849786 11411 heartbeater.cc:347] Connected to a master server at 127.3.246.126:46187
I20260924 07:18:49.849877 11411 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:49.850050 11411 heartbeater.cc:511] Master 127.3.246.126:46187 requested a full tablet report, sending...
I20260924 07:18:49.850412 10167 ts_manager.cc:194] Re-registered known tserver with Master: 16eef29dfa904346bcb8ff1485075d58 (127.3.246.67:33321)
I20260924 07:18:49.850889 10167 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:41921
I20260924 07:18:49.851147 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.835415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:55987 (local address 127.3.246.124:42711)
0924 07:18:49.835459 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.835461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.835473 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:18:49.836115 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.836117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.836132 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.836168 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.837277 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.838049 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.839594 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.839894 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.839933 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.847319 (+ 7386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.847320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.847320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.847328 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.848878 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.850016 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.850017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.850017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.850073 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.850443 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.850444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.850445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.850567 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.850632 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.850948 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.851028 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:18:49.851473 11415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.835021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:42711 (local address 127.3.246.67:55987)
0924 07:18:49.835218 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.835264 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.835972 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.836329 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.836347 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.836840 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.837170 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.837177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.838183 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.838184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.839512 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.839514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.840014 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.840014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.840148 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.840940 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.840945 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.847227 (+ 6282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.849202 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.849203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.849204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.849406 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.850319 (+ 913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.850320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.850320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.850384 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.850778 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.850779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.850822 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.851260 (+ 438us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.851368 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":129,"thread_start_us":95,"threads_started":1}
I20260924 07:18:49.851804 11412 heartbeater.cc:347] Connected to a master server at 127.3.246.124:42711
I20260924 07:18:49.851857 11412 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:49.851919 11412 heartbeater.cc:511] Master 127.3.246.124:42711 requested a full tablet report, sending...
I20260924 07:18:49.851933 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.838128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.3.246.125:35077)
0924 07:18:49.841771 (+ 3643us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.841773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.842344 (+ 571us) server_negotiation.cc:407] Connection header received
0924 07:18:49.842370 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.842371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.842384 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.842438 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.843098 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.843826 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.844982 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.845285 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.845336 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.847108 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.847108 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.847109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.847115 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.850522 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.850895 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.850896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.850897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.850956 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.851146 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.851147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.851147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.851279 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.851370 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.851752 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.851836 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3621,"thread_start_us":61,"threads_started":1}
I20260924 07:18:49.852180 10959 ts_manager.cc:194] Registered new tserver with Master: 16eef29dfa904346bcb8ff1485075d58 (127.3.246.67:33321)
I20260924 07:18:49.852347 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.835742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:38985 (local address 127.3.246.125:35077)
0924 07:18:49.835792 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.835793 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.835884 (+ 91us) server_negotiation.cc:407] Connection header received
0924 07:18:49.836089 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.836091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.836104 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.836138 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.837280 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.838090 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.839616 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.839919 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.839960 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.847694 (+ 7734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.847695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.847695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.847702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.849781 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.850766 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.850767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.850767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.850822 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.851095 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.851096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.851096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.851223 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.851265 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.852168 (+ 903us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.852242 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:18:49.852873 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.835651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35077 (local address 127.3.246.67:38985)
0924 07:18:49.835785 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.835794 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.835985 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.836689 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.836689 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.836867 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:49.837168 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.837174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.838185 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.838185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.839514 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.839517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.840039 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.840039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.840149 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.840862 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:49.840866 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:49.847593 (+ 6727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:49.850494 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.850495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.850495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.850698 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.850898 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:49.850899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:49.850900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:49.851014 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:49.851493 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:49.851493 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:49.851808 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.852625 (+ 817us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.852744 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":80,"thread_start_us":86,"threads_started":1}
I20260924 07:18:49.853183 11413 heartbeater.cc:347] Connected to a master server at 127.3.246.125:35077
I20260924 07:18:49.853230 11413 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:49.853271 11413 heartbeater.cc:511] Master 127.3.246.125:35077 requested a full tablet report, sending...
I20260924 07:18:49.853272 11424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.837663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.3.246.126:46187)
0924 07:18:49.842053 (+ 4390us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.842055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.842060 (+ 5us) server_negotiation.cc:407] Connection header received
0924 07:18:49.842076 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.842076 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.842084 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.842165 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.842960 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.843807 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.845148 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.845437 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.845491 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.846973 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.846974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.846975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.846982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.851943 (+ 4961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.852326 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.852327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.852327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.852385 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.852581 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.852581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.852582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.852700 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.852803 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.852863 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.853153 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4369,"thread_start_us":74,"threads_started":1}
I20260924 07:18:49.853629 10771 ts_manager.cc:194] Registered new tserver with Master: 16eef29dfa904346bcb8ff1485075d58 (127.3.246.67:33321)
I20260924 07:18:49.854648 10167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:49.856181 10167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:49.863320 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.859680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:33321 (local address 127.0.0.1:46748)
0924 07:18:49.859738 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.859750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.859792 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.860281 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.860282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.860298 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:49.860344 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.860346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.861469 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.861470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.862435 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.862437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.863075 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.863077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.863135 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.863139 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.863206 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.863250 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 07:18:49.863337 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.859807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.3.246.67:33321)
0924 07:18:49.860067 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.860073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.860084 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:18:49.860122 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.860125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.860141 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.860424 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:49.860481 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.861383 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.862519 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.862988 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.863078 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.863153 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.863219 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.863253 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":107,"threads_started":1}
I20260924 07:18:49.863899 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.859752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:41413 (local address 127.0.0.1:34604)
0924 07:18:49.859917 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.859928 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.859982 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.860561 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.860562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.860564 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:49.860606 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.860608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.861981 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.861982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.862764 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.862766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.863684 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.863685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.863744 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.863749 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.863778 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.863830 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":20,"thread_start_us":100,"threads_started":1}
I20260924 07:18:49.864241 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.860189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34604 (local address 127.3.246.65:41413)
0924 07:18:49.860368 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.860372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.860381 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:49.860432 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.860434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.860451 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.860684 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:49.860730 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.861881 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.862872 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.863588 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.864011 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.864093 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.864120 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.864160 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260924 07:18:49.864173 11345 tablet_service.cc:1467] Processing CreateTablet for tablet e32cd35e1a3646498c23ada2675fb7a7 (DEFAULT_TABLE table=test-table [id=7adb4aaa68e944db82ab6b0910c418f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:49.864516 11345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32cd35e1a3646498c23ada2675fb7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:49.864861 11072 tablet_service.cc:1467] Processing CreateTablet for tablet e32cd35e1a3646498c23ada2675fb7a7 (DEFAULT_TABLE table=test-table [id=7adb4aaa68e944db82ab6b0910c418f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:49.865159 11072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32cd35e1a3646498c23ada2675fb7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:49.867298 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.860011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:34417 (local address 127.0.0.1:55452)
0924 07:18:49.860176 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.860186 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.860227 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.860801 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.860802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.860804 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:49.860838 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.860839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.862043 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.862044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.866051 (+ 4007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.866054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.867064 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.867065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.867125 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.867130 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.867182 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.867224 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260924 07:18:49.867697 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.860462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55452 (local address 127.3.246.66:34417)
0924 07:18:49.860639 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.860644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.860653 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:49.860690 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.860692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.860708 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.860799 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:49.860924 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.861966 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.866159 (+ 4193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.866885 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.866960 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.867028 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.867564 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.867613 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260924 07:18:49.868022 11438 tablet_bootstrap.cc:492] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86: Bootstrap starting.
I20260924 07:18:49.867985 11437 tablet_bootstrap.cc:492] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58: Bootstrap starting.
I20260924 07:18:49.868425 11209 tablet_service.cc:1467] Processing CreateTablet for tablet e32cd35e1a3646498c23ada2675fb7a7 (DEFAULT_TABLE table=test-table [id=7adb4aaa68e944db82ab6b0910c418f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:49.868707 11437 tablet_bootstrap.cc:654] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:49.868710 11438 tablet_bootstrap.cc:654] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:49.868753 11209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32cd35e1a3646498c23ada2675fb7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:49.869028 11437 log.cc:824] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:49.869055 11438 log.cc:824] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:49.869835 11437 tablet_bootstrap.cc:492] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58: No bootstrap required, opened a new log
I20260924 07:18:49.869917 11437 ts_tablet_manager.cc:1397] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:49.870075 11438 tablet_bootstrap.cc:492] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86: No bootstrap required, opened a new log
I20260924 07:18:49.870152 11438 ts_tablet_manager.cc:1397] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:49.872161 11441 tablet_bootstrap.cc:492] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e: Bootstrap starting.
I20260924 07:18:49.872078 11437 raft_consensus.cc:348] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } }
I20260924 07:18:49.872241 11437 raft_consensus.cc:374] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:49.872279 11437 raft_consensus.cc:729] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16eef29dfa904346bcb8ff1485075d58, State: Initialized, Role: FOLLOWER
I20260924 07:18:49.872205 11438 raft_consensus.cc:348] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } }
I20260924 07:18:49.872419 11438 raft_consensus.cc:374] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:49.872476 11438 raft_consensus.cc:729] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be36e4202744663900a72eb10a5ae86, State: Initialized, Role: FOLLOWER
I20260924 07:18:49.872411 11437 consensus_queue.cc:260] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } }
I20260924 07:18:49.872598 11438 consensus_queue.cc:260] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } }
I20260924 07:18:49.872764 11437 ts_tablet_manager.cc:1428] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:49.872859 11438 ts_tablet_manager.cc:1428] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:49.872891 11411 heartbeater.cc:503] Master 127.3.246.126:46187 was elected leader, sending a full tablet report...
I20260924 07:18:49.873013 11138 heartbeater.cc:503] Master 127.3.246.126:46187 was elected leader, sending a full tablet report...
I20260924 07:18:49.873090 11441 tablet_bootstrap.cc:654] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:49.873507 11441 log.cc:824] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:49.875070 11441 tablet_bootstrap.cc:492] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e: No bootstrap required, opened a new log
I20260924 07:18:49.875136 11441 ts_tablet_manager.cc:1397] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 07:18:49.877063 11441 raft_consensus.cc:348] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } }
I20260924 07:18:49.877183 11441 raft_consensus.cc:374] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:49.877205 11441 raft_consensus.cc:729] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf13e4ca3f8c4b08bedf0e3654c3e86e, State: Initialized, Role: FOLLOWER
I20260924 07:18:49.877338 11441 consensus_queue.cc:260] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } }
I20260924 07:18:49.877601 11441 ts_tablet_manager.cc:1428] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 07:18:49.877700 11275 heartbeater.cc:503] Master 127.3.246.126:46187 was elected leader, sending a full tablet report...
I20260924 07:18:49.911572 11443 raft_consensus.cc:482] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:49.911758 11443 raft_consensus.cc:504] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } }
I20260924 07:18:49.912161 11443 leader_election.cc:288] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16eef29dfa904346bcb8ff1485075d58 (127.3.246.67:33321), bf13e4ca3f8c4b08bedf0e3654c3e86e (127.3.246.66:34417)
I20260924 07:18:49.915338 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.912532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:56345 (local address 127.3.246.67:33321)
0924 07:18:49.912585 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.912587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.912597 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:18:49.912628 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.912629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.912639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.912725 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:49.912908 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.913467 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.914597 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.914942 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.915064 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.915072 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.915230 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.915269 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:18:49.915346 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.912546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:51765 (local address 127.3.246.66:34417)
0924 07:18:49.912594 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.912596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.912701 (+ 105us) server_negotiation.cc:407] Connection header received
0924 07:18:49.912751 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.912752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.912762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.912820 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:49.912959 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.913465 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.914588 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.914966 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.915060 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.915069 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.915235 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.915274 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:18:49.915391 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.912426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:34417 (local address 127.3.246.65:51765)
0924 07:18:49.912635 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.912644 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.912692 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.912822 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.912823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.912824 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:49.912888 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.912890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.913553 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.913554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.914485 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.914488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.915059 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.915060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.915138 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.915141 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.915225 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.915263 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":230}
I20260924 07:18:49.915405 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.912420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:33321 (local address 127.3.246.65:56345)
0924 07:18:49.912482 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.912495 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.912555 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.912748 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.912749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.912751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:49.912826 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.912828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.913540 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.913541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.914484 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.914487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.915055 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.915056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.915130 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.915134 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.915221 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.915261 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":68}
I20260924 07:18:49.915740 11365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32cd35e1a3646498c23ada2675fb7a7" candidate_uuid: "7be36e4202744663900a72eb10a5ae86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eef29dfa904346bcb8ff1485075d58" is_pre_election: true
I20260924 07:18:49.915735 11229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32cd35e1a3646498c23ada2675fb7a7" candidate_uuid: "7be36e4202744663900a72eb10a5ae86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" is_pre_election: true
I20260924 07:18:49.915894 11365 raft_consensus.cc:2454] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7be36e4202744663900a72eb10a5ae86 in term 0.
I20260924 07:18:49.915894 11229 raft_consensus.cc:2454] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7be36e4202744663900a72eb10a5ae86 in term 0.
I20260924 07:18:49.916136 11027 leader_election.cc:302] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7be36e4202744663900a72eb10a5ae86, bf13e4ca3f8c4b08bedf0e3654c3e86e; no voters:
I20260924 07:18:49.916319 11443 raft_consensus.cc:2781] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:49.916358 11443 raft_consensus.cc:482] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:49.916373 11443 raft_consensus.cc:3037] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:49.917192 11443 raft_consensus.cc:504] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } }
I20260924 07:18:49.917335 11443 leader_election.cc:288] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [CANDIDATE]: Term 1 election: Requested vote from peers 16eef29dfa904346bcb8ff1485075d58 (127.3.246.67:33321), bf13e4ca3f8c4b08bedf0e3654c3e86e (127.3.246.66:34417)
I20260924 07:18:49.917459 11365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32cd35e1a3646498c23ada2675fb7a7" candidate_uuid: "7be36e4202744663900a72eb10a5ae86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eef29dfa904346bcb8ff1485075d58"
I20260924 07:18:49.917464 11229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32cd35e1a3646498c23ada2675fb7a7" candidate_uuid: "7be36e4202744663900a72eb10a5ae86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e"
I20260924 07:18:49.917542 11229 raft_consensus.cc:3037] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:49.917560 11365 raft_consensus.cc:3037] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:49.918291 11365 raft_consensus.cc:2454] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7be36e4202744663900a72eb10a5ae86 in term 1.
I20260924 07:18:49.918339 11229 raft_consensus.cc:2454] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7be36e4202744663900a72eb10a5ae86 in term 1.
I20260924 07:18:49.918499 11026 leader_election.cc:302] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16eef29dfa904346bcb8ff1485075d58, 7be36e4202744663900a72eb10a5ae86; no voters:
I20260924 07:18:49.918597 11443 raft_consensus.cc:2781] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:49.918754 11443 raft_consensus.cc:686] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [term 1 LEADER]: Becoming Leader. State: Replica: 7be36e4202744663900a72eb10a5ae86, State: Running, Role: LEADER
I20260924 07:18:49.918871 11443 consensus_queue.cc:237] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } }
I20260924 07:18:49.919827 10167 catalog_manager.cc:5495] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7be36e4202744663900a72eb10a5ae86 (127.3.246.65). New cstate: current_term: 1 leader_uuid: "7be36e4202744663900a72eb10a5ae86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } health_report { overall_health: UNKNOWN } } }
I20260924 07:18:49.925534 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.923297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.3.246.124:42711)
0924 07:18:49.923363 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.923365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.923372 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:18:49.923396 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.923397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.923414 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.923467 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:49.923564 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.924035 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.924909 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.925290 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.925357 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.925403 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.925459 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.925486 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:18:49.925540 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.923200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:42711 (local address 127.0.0.1:45682)
0924 07:18:49.923271 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:49.923284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:49.923336 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:49.923481 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:49.923482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:49.923483 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:49.923524 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.923525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.924097 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.924097 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.924786 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:49.924787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.925366 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.925367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:49.925411 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.925414 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:49.925455 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 07:18:49.925489 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 07:18:49.925889 10969 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.10] Dedup: 1.7->[1.8-1.10]
I20260924 07:18:49.926787 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4f9ee1f206c43caa1a7f55aae00a8ab. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } }
I20260924 07:18:49.926898 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: This master's current role is: LEARNER
I20260924 07:18:49.935714 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.933482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.3.246.65:41413)
0924 07:18:49.933540 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.933542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.933568 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:18:49.933606 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.933607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.933623 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.933716 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:49.933873 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.934351 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.935053 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.935228 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.935276 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.935420 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:49.935511 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:49.935595 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.935632 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.935650 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:18:49.937594 11229 raft_consensus.cc:1261] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 1 FOLLOWER]: Refusing update from remote peer 7be36e4202744663900a72eb10a5ae86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:49.938004 11365 raft_consensus.cc:1261] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58 [term 1 FOLLOWER]: Refusing update from remote peer 7be36e4202744663900a72eb10a5ae86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:49.938074 11446 consensus_queue.cc:1035] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:49.938330 11446 consensus_queue.cc:1035] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:49.940496 11452 mvcc.cc:205] Tried to move back new op lower bound from 7332799815421304832 to 7332799815348342784. Current Snapshot: MvccSnapshot[applied={T|T < 7332799815421304832}]
I20260924 07:18:49.941334 11453 mvcc.cc:205] Tried to move back new op lower bound from 7332799815421304832 to 7332799815348342784. Current Snapshot: MvccSnapshot[applied={T|T < 7332799815421304832}]
I20260924 07:18:49.941866 11454 mvcc.cc:205] Tried to move back new op lower bound from 7332799815421304832 to 7332799815348342784. Current Snapshot: MvccSnapshot[applied={T|T < 7332799815421304832}]
I20260924 07:18:49.942992 10167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:49.943082 10167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:49.943619 10726 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } }
I20260924 07:18:49.943739 10726 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:49.944610 10919 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab: attempting to promote NON_VOTER 5738c38db05a44ea978f5e4fed86171f to VOTER
I20260924 07:18:49.944728 10919 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } }
I20260924 07:18:49.945075 10781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 FOLLOWER]: Refusing update from remote peer e4f9ee1f206c43caa1a7f55aae00a8ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260924 07:18:49.945066 10167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab: Sending DeleteTablet for 3 replicas of tablet e32cd35e1a3646498c23ada2675fb7a7
I20260924 07:18:49.945257 10919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260924 07:18:49.945482 11345 tablet_service.cc:1514] Processing DeleteTablet for tablet e32cd35e1a3646498c23ada2675fb7a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:49 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:46748
I20260924 07:18:49.945711 11462 tablet_replica.cc:330] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58: stopping tablet replica
I20260924 07:18:49.945703 11209 tablet_service.cc:1514] Processing DeleteTablet for tablet e32cd35e1a3646498c23ada2675fb7a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:49 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:55452
I20260924 07:18:49.945796 11462 raft_consensus.cc:2229] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:49.945823 11072 tablet_service.cc:1514] Processing DeleteTablet for tablet e32cd35e1a3646498c23ada2675fb7a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:49 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:34604
I20260924 07:18:49.945888 11463 tablet_replica.cc:330] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e: stopping tablet replica
I20260924 07:18:49.945897 11462 raft_consensus.cc:2258] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:49.945981 11463 raft_consensus.cc:2229] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:49.946071 11463 raft_consensus.cc:2258] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:49.946321 11462 ts_tablet_manager.cc:1905] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:49.946400 10969 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 1 LEARNER]: Refusing update from remote peer e4f9ee1f206c43caa1a7f55aae00a8ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260924 07:18:49.946743 10727 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260924 07:18:49.947384 10726 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4f9ee1f206c43caa1a7f55aae00a8ab [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 5738c38db05a44ea978f5e4fed86171f (127.3.246.124) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } } }
I20260924 07:18:49.947552 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 10135
I20260924 07:18:49.948151 11462 ts_tablet_manager.cc:1918] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:49.948222 11462 log.cc:1196] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/e32cd35e1a3646498c23ada2675fb7a7
I20260924 07:18:49.948475 11462 ts_tablet_manager.cc:1939] T e32cd35e1a3646498c23ada2675fb7a7 P 16eef29dfa904346bcb8ff1485075d58: Deleting consensus metadata
W20260924 07:18:49.953307 10755 connection.cc:535] server connection from 127.0.0.1:35254 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:18:49.953501 11464 tablet_replica.cc:330] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86: stopping tablet replica
I20260924 07:18:49.953572 11463 ts_tablet_manager.cc:1905] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:18:49.953593 11300 connection.cc:535] server connection from 127.0.0.1:46748 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:18:49.953917 11464 raft_consensus.cc:2229] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:49.954912 11464 raft_consensus.cc:2258] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:49.955400 11463 ts_tablet_manager.cc:1918] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:49.955476 11463 log.cc:1196] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/e32cd35e1a3646498c23ada2675fb7a7
I20260924 07:18:49.955508 11464 ts_tablet_manager.cc:1905] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:49.955787 11463 ts_tablet_manager.cc:1939] T e32cd35e1a3646498c23ada2675fb7a7 P bf13e4ca3f8c4b08bedf0e3654c3e86e: Deleting consensus metadata
W20260924 07:18:49.956159 11162 connection.cc:441] server connection from 127.0.0.1:55452 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55452 (request call id 1) could send its response
I20260924 07:18:49.957016 11464 ts_tablet_manager.cc:1918] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:49.957093 11464 log.cc:1196] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-0/wal/wals/e32cd35e1a3646498c23ada2675fb7a7
I20260924 07:18:49.957355 11464 ts_tablet_manager.cc:1939] T e32cd35e1a3646498c23ada2675fb7a7 P 7be36e4202744663900a72eb10a5ae86: Deleting consensus metadata
W20260924 07:18:49.957643 11027 connection.cc:441] server connection from 127.0.0.1:34604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34604 (request call id 1) could send its response
I20260924 07:18:49.963129 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.946997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45694 (local address 127.3.246.124:42711)
0924 07:18:49.947037 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.947040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.953771 (+ 6731us) server_negotiation.cc:407] Connection header received
0924 07:18:49.954046 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.954047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.954059 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.954091 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.955103 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.955853 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.957146 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.957456 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.957503 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.959478 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.959480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.959480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.959486 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.961112 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.961476 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.961477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.961478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.961532 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.962517 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.962518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.962518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.962665 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.962748 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.962953 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.963022 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:18:49.963481 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.955281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45696 (local address 127.3.246.124:42711)
0924 07:18:49.955340 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.955342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.955619 (+ 277us) server_negotiation.cc:407] Connection header received
0924 07:18:49.955719 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.955720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.955733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.955762 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.956690 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.957462 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.958541 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.958830 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.958869 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.960705 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.960707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.960707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.960714 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.962268 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.962669 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.962670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.962670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.962715 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.962920 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.962920 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.962921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.963027 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.963088 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.963350 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.963428 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:18:49.965015 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.956491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.3.246.125:35077)
0924 07:18:49.956530 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.956531 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.956819 (+ 288us) server_negotiation.cc:407] Connection header received
0924 07:18:49.956909 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.956910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.956920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.956948 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.957612 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.958372 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.959349 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.959637 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.959712 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.961933 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.961934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.961935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.961942 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.963632 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.964227 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.964228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.964229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.964291 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.964515 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.964516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.964517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.964630 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.964731 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.964835 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.964912 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:18:50.943281 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:50.942915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.65:56921)
0924 07:18:50.943114 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:50.943215 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20260924 07:18:50.943702 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:50.943452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.66:33815)
0924 07:18:50.943554 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:50.943655 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":57,"threads_started":1}
W20260924 07:18:50.944001 11275 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:46187 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
I20260924 07:18:50.944000 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:50.943770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.67:40651)
0924 07:18:50.943897 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:50.943960 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":49,"threads_started":1}
W20260924 07:18:50.944015 11138 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:46187 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
W20260924 07:18:50.944219 11411 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:46187 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
I20260924 07:18:51.369904 11476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4f9ee1f206c43caa1a7f55aae00a8ab)
I20260924 07:18:51.370118 11476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } }
I20260924 07:18:51.370416 11476 leader_election.cc:288] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4f9ee1f206c43caa1a7f55aae00a8ab (127.3.246.126:46187), 13af89bf44fa42fe9c5a301d79df7ac2 (127.3.246.125:35077)
I20260924 07:18:51.370819 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.370608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:32950)
0924 07:18:51.370716 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.370771 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":60,"threads_started":1}
W20260924 07:18:51.370945 10943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
I20260924 07:18:51.371431 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.371316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:32956)
0924 07:18:51.371373 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.371408 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 07:18:51.371531 10943 leader_election.cc:334] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e4f9ee1f206c43caa1a7f55aae00a8ab (127.3.246.126:46187): Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
I20260924 07:18:51.373576 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.370808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35077 (local address 127.0.0.1:35280)
0924 07:18:51.370891 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.370900 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.370962 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.371184 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.371185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.371189 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:51.371245 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.371246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.372011 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.372012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.372768 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.372770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.373342 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.373343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.373391 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.373395 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.373464 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.373510 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":58,"threads_started":1}
I20260924 07:18:51.373587 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.370899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.3.246.125:35077)
0924 07:18:51.371011 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.371014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.371025 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:18:51.371060 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.371061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.371073 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.371153 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:51.371337 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.371940 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.372847 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.373268 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.373330 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.373339 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.373473 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.373523 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260924 07:18:51.373898 10781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5738c38db05a44ea978f5e4fed86171f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" is_pre_election: true
I20260924 07:18:51.374228 10943 leader_election.cc:302] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5738c38db05a44ea978f5e4fed86171f; no voters: 13af89bf44fa42fe9c5a301d79df7ac2, e4f9ee1f206c43caa1a7f55aae00a8ab
I20260924 07:18:51.374387 11476 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:18:51.446185 11481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4f9ee1f206c43caa1a7f55aae00a8ab)
I20260924 07:18:51.446331 11481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } }
I20260924 07:18:51.446630 11481 leader_election.cc:288] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4f9ee1f206c43caa1a7f55aae00a8ab (127.3.246.126:46187), 5738c38db05a44ea978f5e4fed86171f (127.3.246.124:42711)
I20260924 07:18:51.446955 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.446703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:32990)
0924 07:18:51.446844 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.446913 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
W20260924 07:18:51.447072 10755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
I20260924 07:18:51.447526 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.447433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:33004)
0924 07:18:51.447478 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.447505 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 07:18:51.447646 10755 leader_election.cc:334] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e4f9ee1f206c43caa1a7f55aae00a8ab (127.3.246.126:46187): Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
I20260924 07:18:51.449565 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.446802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:42711 (local address 127.0.0.1:45708)
0924 07:18:51.446905 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.446915 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.446966 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.447235 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.447236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.447239 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:51.447301 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.447303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.448098 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.448099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.448875 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.448877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.449389 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.449390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.449438 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.449442 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.449486 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.449520 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":43,"threads_started":1}
I20260924 07:18:51.449584 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.446926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.3.246.124:42711)
0924 07:18:51.447076 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.447079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.447088 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:51.447116 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.447117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.447128 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.447219 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:51.447385 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.448019 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.448949 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.449316 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.449376 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.449384 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.449504 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.449532 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260924 07:18:51.449815 10969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5738c38db05a44ea978f5e4fed86171f" is_pre_election: true
I20260924 07:18:51.449928 10969 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13af89bf44fa42fe9c5a301d79df7ac2 in term 1.
I20260924 07:18:51.450101 10758 leader_election.cc:302] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13af89bf44fa42fe9c5a301d79df7ac2, 5738c38db05a44ea978f5e4fed86171f; no voters: e4f9ee1f206c43caa1a7f55aae00a8ab
I20260924 07:18:51.450214 11481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 07:18:51.450251 11481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4f9ee1f206c43caa1a7f55aae00a8ab)
I20260924 07:18:51.450270 11481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:18:51.450981 11481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } }
I20260924 07:18:51.451124 11481 leader_election.cc:288] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [CANDIDATE]: Term 2 election: Requested vote from peers e4f9ee1f206c43caa1a7f55aae00a8ab (127.3.246.126:46187), 5738c38db05a44ea978f5e4fed86171f (127.3.246.124:42711)
I20260924 07:18:51.451269 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.451189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:33016)
0924 07:18:51.451226 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.451246 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 07:18:51.451297 10969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5738c38db05a44ea978f5e4fed86171f"
I20260924 07:18:51.451402 10969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:18:51.451623 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.451531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:33020)
0924 07:18:51.451579 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.451603 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 07:18:51.451745 10755 leader_election.cc:334] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e4f9ee1f206c43caa1a7f55aae00a8ab (127.3.246.126:46187): Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
I20260924 07:18:51.452150 10969 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13af89bf44fa42fe9c5a301d79df7ac2 in term 2.
I20260924 07:18:51.452322 10758 leader_election.cc:302] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13af89bf44fa42fe9c5a301d79df7ac2, 5738c38db05a44ea978f5e4fed86171f; no voters: e4f9ee1f206c43caa1a7f55aae00a8ab
I20260924 07:18:51.452409 11481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:18:51.452535 11481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 2 LEADER]: Becoming Leader. State: Replica: 13af89bf44fa42fe9c5a301d79df7ac2, State: Running, Role: LEADER
I20260924 07:18:51.452615 11481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } }
I20260924 07:18:51.452996 11486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13af89bf44fa42fe9c5a301d79df7ac2. Latest consensus state: current_term: 2 leader_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } } pending_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } } }
I20260924 07:18:51.453114 11486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:51.453316 11488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:51.453728 11488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:51.453845 11488 catalog_manager.cc:1259] Loaded cluster ID: f5a9a963570944cc95a964bb074d0315
I20260924 07:18:51.453894 11488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:51.454063 11488 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:51.454154 11488 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: Loaded TSK: 0
I20260924 07:18:51.454301 11488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:18:51.508991 10770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:51.510673 10770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:51.511662 10969 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 2 FOLLOWER]: Refusing update from remote peer 13af89bf44fa42fe9c5a301d79df7ac2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 07:18:51.511665 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.511517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:33040)
0924 07:18:51.511593 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.511632 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 07:18:51.512032 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.511930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.0.0.1:33042)
0924 07:18:51.511977 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.512000 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 07:18:51.511973 11486 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
W20260924 07:18:51.512151 10755 consensus_peers.cc:487] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 -> Peer e4f9ee1f206c43caa1a7f55aae00a8ab (127.3.246.126:46187): Couldn't send request to peer e4f9ee1f206c43caa1a7f55aae00a8ab. Status: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 07:18:51.512717 11476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13af89bf44fa42fe9c5a301d79df7ac2. Latest consensus state: current_term: 2 leader_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: NON_VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: true } } } pending_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } } }
I20260924 07:18:51.512820 11476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:18:51.513173 11481 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [term 2 LEADER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 5738c38db05a44ea978f5e4fed86171f (127.3.246.124) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } } }
I20260924 07:18:51.513289 10969 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [term 2 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 5738c38db05a44ea978f5e4fed86171f (127.3.246.124) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } } }
I20260924 07:18:51.514391 11486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } } }
I20260924 07:18:51.514400 11481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } } }
I20260924 07:18:51.514489 11486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:51.514511 11481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:51.514474 11476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } } }
I20260924 07:18:51.514549 11476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:18:51.514449 11494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e4f9ee1f206c43caa1a7f55aae00a8ab" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46187 } } peers { permanent_uuid: "13af89bf44fa42fe9c5a301d79df7ac2" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35077 } attrs { promote: false } } peers { permanent_uuid: "5738c38db05a44ea978f5e4fed86171f" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 42711 } attrs { promote: false } } }
I20260924 07:18:51.514704 11494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5738c38db05a44ea978f5e4fed86171f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:18:51.514775 11490 mvcc.cc:205] Tried to move back new op lower bound from 7332799821869719552 to 7332799821630664704. Current Snapshot: MvccSnapshot[applied={T|T < 7332799821869719552}]
I20260924 07:18:51.519695 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.516344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:33321 (local address 127.0.0.1:46760)
0924 07:18:51.516515 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.516525 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.516587 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.516744 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.516745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.516746 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:51.516798 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.516800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.517851 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.517851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.518690 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.518692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.519476 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.519477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.519537 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.519543 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.519571 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.519610 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":1}
I20260924 07:18:51.519937 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.516344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:34417 (local address 127.0.0.1:55462)
0924 07:18:51.516574 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.516584 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.516632 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.516813 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.516815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.516818 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:51.516866 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.516868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.517697 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.517698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.518685 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.518687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.519721 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.519722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.519785 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.519788 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.519835 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.519875 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":202}
I20260924 07:18:51.519979 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.516424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.3.246.66:34417)
0924 07:18:51.516548 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.516551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.516685 (+ 134us) server_negotiation.cc:407] Connection header received
0924 07:18:51.516720 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.516721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.516745 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.516809 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:51.516943 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.517610 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.518935 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.519636 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.519714 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.519728 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.519855 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.519904 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260924 07:18:51.520025 11497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.516408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.3.246.67:33321)
0924 07:18:51.516610 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.516612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.516620 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:18:51.516649 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.516649 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.516662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.516737 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:51.516959 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.517761 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.518809 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.519387 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.519871 (+ 484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.519884 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.519903 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.519942 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260924 07:18:51.520242 11345 tablet_service.cc:1467] Processing CreateTablet for tablet 2668615952424cd2b9f8c26d0056d7b4 (DEFAULT_TABLE table=test-table [id=0982af4e9c5f42ce92579624ebd51049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:51.520304 11209 tablet_service.cc:1467] Processing CreateTablet for tablet 2668615952424cd2b9f8c26d0056d7b4 (DEFAULT_TABLE table=test-table [id=0982af4e9c5f42ce92579624ebd51049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:51.520344 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.516630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:41413 (local address 127.0.0.1:34624)
0924 07:18:51.516886 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.516894 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.516936 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.517290 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.517291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.517293 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:51.517329 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.517331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.518202 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.518203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.519161 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.519163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.520102 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.520103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.520165 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.520170 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.520230 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.520273 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":79,"threads_started":1}
I20260924 07:18:51.520373 11345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2668615952424cd2b9f8c26d0056d7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:51.520383 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.516807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.3.246.65:41413)
0924 07:18:51.517060 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.517164 (+ 104us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.517172 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:18:51.517197 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.517198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.517210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.517270 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:51.517401 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.518120 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.519248 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.519639 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.519738 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.519751 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.520265 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.520316 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":67,"threads_started":1}
I20260924 07:18:51.520422 11209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2668615952424cd2b9f8c26d0056d7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:51.521502 11500 tablet_bootstrap.cc:492] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58: Bootstrap starting.
I20260924 07:18:51.521600 11501 tablet_bootstrap.cc:492] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e: Bootstrap starting.
I20260924 07:18:51.521737 11072 tablet_service.cc:1467] Processing CreateTablet for tablet 2668615952424cd2b9f8c26d0056d7b4 (DEFAULT_TABLE table=test-table [id=0982af4e9c5f42ce92579624ebd51049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:51.521842 11072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2668615952424cd2b9f8c26d0056d7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:51.521898 11500 tablet_bootstrap.cc:654] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:51.521945 11501 tablet_bootstrap.cc:654] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:51.522442 11501 tablet_bootstrap.cc:492] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e: No bootstrap required, opened a new log
I20260924 07:18:51.522508 11501 ts_tablet_manager.cc:1397] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:51.522675 11500 tablet_bootstrap.cc:492] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58: No bootstrap required, opened a new log
I20260924 07:18:51.522642 11501 raft_consensus.cc:348] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } }
I20260924 07:18:51.522720 11501 raft_consensus.cc:374] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:51.522714 11500 ts_tablet_manager.cc:1397] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:51.522743 11501 raft_consensus.cc:729] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf13e4ca3f8c4b08bedf0e3654c3e86e, State: Initialized, Role: FOLLOWER
I20260924 07:18:51.522799 11501 consensus_queue.cc:260] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } }
I20260924 07:18:51.522832 11500 raft_consensus.cc:348] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } }
I20260924 07:18:51.522899 11500 raft_consensus.cc:374] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:51.522921 11500 raft_consensus.cc:729] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16eef29dfa904346bcb8ff1485075d58, State: Initialized, Role: FOLLOWER
I20260924 07:18:51.522936 11501 ts_tablet_manager.cc:1428] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:51.522969 11500 consensus_queue.cc:260] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } }
I20260924 07:18:51.523120 11500 ts_tablet_manager.cc:1428] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:51.523540 11508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.523310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.66:54901)
0924 07:18:51.523453 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.523495 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260924 07:18:51.523703 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.523310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.67:33911)
0924 07:18:51.523635 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.523672 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":86,"threads_started":1}
I20260924 07:18:51.524117 11508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.524031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.66:56701)
0924 07:18:51.524068 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.524092 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 07:18:51.524340 11413 heartbeater.cc:503] Master 127.3.246.125:35077 was elected leader, sending a full tablet report...
I20260924 07:18:51.524498 11506 tablet_bootstrap.cc:492] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86: Bootstrap starting.
I20260924 07:18:51.524523 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.524416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.67:56167)
0924 07:18:51.524466 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.524493 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 07:18:51.524873 11506 tablet_bootstrap.cc:654] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:51.525372 11506 tablet_bootstrap.cc:492] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86: No bootstrap required, opened a new log
I20260924 07:18:51.525411 11506 ts_tablet_manager.cc:1397] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:51.525520 11506 raft_consensus.cc:348] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } }
I20260924 07:18:51.525583 11506 raft_consensus.cc:374] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:51.525597 11506 raft_consensus.cc:729] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be36e4202744663900a72eb10a5ae86, State: Initialized, Role: FOLLOWER
I20260924 07:18:51.525656 11506 consensus_queue.cc:260] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } }
I20260924 07:18:51.525970 11506 ts_tablet_manager.cc:1428] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:51.526685 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.526370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.65:50791)
0924 07:18:51.526579 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.526618 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":101,"threads_started":1}
I20260924 07:18:51.527014 11140 heartbeater.cc:503] Master 127.3.246.125:35077 was elected leader, sending a full tablet report...
I20260924 07:18:51.527379 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.527246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.65:36211)
0924 07:18:51.527315 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.527346 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 07:18:51.559206 11504 raft_consensus.cc:482] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:51.559325 11504 raft_consensus.cc:504] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } }
I20260924 07:18:51.559703 11504 leader_election.cc:288] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16eef29dfa904346bcb8ff1485075d58 (127.3.246.67:33321), 7be36e4202744663900a72eb10a5ae86 (127.3.246.65:41413)
I20260924 07:18:51.563584 11497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.559891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:35335 (local address 127.3.246.67:33321)
0924 07:18:51.559941 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.559943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.559955 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:18:51.560054 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.560055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.560066 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.560132 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:51.560432 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.561218 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.562842 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.563217 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.563308 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.563320 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.563478 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.563519 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:18:51.563584 11508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.559816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:33321 (local address 127.3.246.66:35335)
0924 07:18:51.559890 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.559903 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.559968 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.560162 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.560163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.560166 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:51.560338 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.560343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.561312 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.561313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.562723 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.562724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.563306 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.563307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.563386 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.563389 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.563466 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.563511 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":45}
I20260924 07:18:51.563647 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.560006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:59753 (local address 127.3.246.65:41413)
0924 07:18:51.560059 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.560061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.560207 (+ 146us) server_negotiation.cc:407] Connection header received
0924 07:18:51.560234 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.560235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.560242 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.560285 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:51.560513 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.561215 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.562804 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.563236 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.563351 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.563362 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.563543 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.563584 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:18:51.563843 11365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2668615952424cd2b9f8c26d0056d7b4" candidate_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eef29dfa904346bcb8ff1485075d58" is_pre_election: true
I20260924 07:18:51.563906 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.559865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:41413 (local address 127.3.246.66:59753)
0924 07:18:51.560088 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.560096 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.560187 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.560359 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.560360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.560363 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:51.560435 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.560436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.561314 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.561314 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.562711 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.562714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.563374 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.563375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.563452 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.563456 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.563531 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.563573 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":85,"thread_start_us":133,"threads_started":1}
I20260924 07:18:51.563942 11365 raft_consensus.cc:2454] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf13e4ca3f8c4b08bedf0e3654c3e86e in term 0.
I20260924 07:18:51.564142 11092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2668615952424cd2b9f8c26d0056d7b4" candidate_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7be36e4202744663900a72eb10a5ae86" is_pre_election: true
I20260924 07:18:51.564188 11163 leader_election.cc:302] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16eef29dfa904346bcb8ff1485075d58, bf13e4ca3f8c4b08bedf0e3654c3e86e; no voters:
I20260924 07:18:51.564280 11092 raft_consensus.cc:2454] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf13e4ca3f8c4b08bedf0e3654c3e86e in term 0.
I20260924 07:18:51.564307 11504 raft_consensus.cc:2781] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:51.564335 11504 raft_consensus.cc:482] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:51.564355 11504 raft_consensus.cc:3037] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 0 FOLLOWER]: Advancing to term 1
W20260924 07:18:51.564338 11141 tablet.cc:2369] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:18:51.565045 11504 raft_consensus.cc:504] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } }
I20260924 07:18:51.565191 11504 leader_election.cc:288] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [CANDIDATE]: Term 1 election: Requested vote from peers 16eef29dfa904346bcb8ff1485075d58 (127.3.246.67:33321), 7be36e4202744663900a72eb10a5ae86 (127.3.246.65:41413)
I20260924 07:18:51.565351 11092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2668615952424cd2b9f8c26d0056d7b4" candidate_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7be36e4202744663900a72eb10a5ae86"
I20260924 07:18:51.565353 11365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2668615952424cd2b9f8c26d0056d7b4" candidate_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eef29dfa904346bcb8ff1485075d58"
I20260924 07:18:51.565428 11365 raft_consensus.cc:3037] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:51.565428 11092 raft_consensus.cc:3037] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:51.566118 11092 raft_consensus.cc:2454] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf13e4ca3f8c4b08bedf0e3654c3e86e in term 1.
I20260924 07:18:51.566269 11365 raft_consensus.cc:2454] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf13e4ca3f8c4b08bedf0e3654c3e86e in term 1.
I20260924 07:18:51.566361 11164 leader_election.cc:302] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7be36e4202744663900a72eb10a5ae86, bf13e4ca3f8c4b08bedf0e3654c3e86e; no voters:
I20260924 07:18:51.566517 11504 raft_consensus.cc:2781] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:51.566710 11504 raft_consensus.cc:686] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 1 LEADER]: Becoming Leader. State: Replica: bf13e4ca3f8c4b08bedf0e3654c3e86e, State: Running, Role: LEADER
I20260924 07:18:51.566773 11504 consensus_queue.cc:237] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } }
I20260924 07:18:51.566843 11277 heartbeater.cc:503] Master 127.3.246.125:35077 was elected leader, sending a full tablet report...
I20260924 07:18:51.567097 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.567005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.66:47355)
0924 07:18:51.567046 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.567074 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 07:18:51.567644 10770 catalog_manager.cc:5495] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e reported cstate change: term changed from 0 to 1, leader changed from <none> to bf13e4ca3f8c4b08bedf0e3654c3e86e (127.3.246.66). New cstate: current_term: 1 leader_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf13e4ca3f8c4b08bedf0e3654c3e86e" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 34417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 } health_report { overall_health: UNKNOWN } } }
I20260924 07:18:51.578789 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.576241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55474 (local address 127.3.246.66:34417)
0924 07:18:51.576284 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.576286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.576295 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:51.576353 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.576354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.576369 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.576433 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:51.576517 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.576982 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.577946 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.578153 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.578214 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.578412 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:51.578547 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:51.578628 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.578676 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.578701 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:18:51.579957 11365 raft_consensus.cc:1261] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58 [term 1 FOLLOWER]: Refusing update from remote peer bf13e4ca3f8c4b08bedf0e3654c3e86e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:51.580106 11092 raft_consensus.cc:1261] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86 [term 1 FOLLOWER]: Refusing update from remote peer bf13e4ca3f8c4b08bedf0e3654c3e86e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:51.580230 11513 consensus_queue.cc:1035] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "16eef29dfa904346bcb8ff1485075d58" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 33321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:51.580353 11504 consensus_queue.cc:1035] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7be36e4202744663900a72eb10a5ae86" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 41413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:51.581202 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.581095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.65:58773)
0924 07:18:51.581139 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.581171 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 07:18:51.581449 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.581038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.67:34645)
0924 07:18:51.581379 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.581413 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320}
I20260924 07:18:51.582620 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.582496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.66:45041)
0924 07:18:51.582564 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.582591 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 07:18:51.582718 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.582435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.65:34415)
0924 07:18:51.582659 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.582689 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
I20260924 07:18:51.582850 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.582596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46187 (local address 127.3.246.67:46519)
0924 07:18:51.582796 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.582824 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:46187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260924 07:18:51.585004 10770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:51.585104 10770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:51.586400 10770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13af89bf44fa42fe9c5a301d79df7ac2: Sending DeleteTablet for 3 replicas of tablet 2668615952424cd2b9f8c26d0056d7b4
I20260924 07:18:51.586643 11345 tablet_service.cc:1514] Processing DeleteTablet for tablet 2668615952424cd2b9f8c26d0056d7b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:51 UTC) from {username='oryx', principal='oryx/127.3.246.125@KRBTEST.COM'} at 127.0.0.1:46760
I20260924 07:18:51.586752 11072 tablet_service.cc:1514] Processing DeleteTablet for tablet 2668615952424cd2b9f8c26d0056d7b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:51 UTC) from {username='oryx', principal='oryx/127.3.246.125@KRBTEST.COM'} at 127.0.0.1:34624
I20260924 07:18:51.587035 11209 tablet_service.cc:1514] Processing DeleteTablet for tablet 2668615952424cd2b9f8c26d0056d7b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:51 UTC) from {username='oryx', principal='oryx/127.3.246.125@KRBTEST.COM'} at 127.0.0.1:55462
I20260924 07:18:51.587062 11526 tablet_replica.cc:330] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86: stopping tablet replica
I20260924 07:18:51.587121 11526 raft_consensus.cc:2229] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:51.587172 11526 raft_consensus.cc:2258] T 2668615952424cd2b9f8c26d0056d7b4 P 7be36e4202744663900a72eb10a5ae86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:51.587306 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 10921
I20260924 07:18:51.594074 11527 tablet_replica.cc:330] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e: stopping tablet replica
I20260924 07:18:51.594151 11527 raft_consensus.cc:2229] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:51.594172 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.594051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:41413 (local address 127.0.0.1:34632)
0924 07:18:51.594104 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.594128 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 07:18:51.594247 11527 raft_consensus.cc:2258] T 2668615952424cd2b9f8c26d0056d7b4 P bf13e4ca3f8c4b08bedf0e3654c3e86e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:51.594365 11525 tablet_replica.cc:330] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58: stopping tablet replica
I20260924 07:18:51.594362 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 11146
I20260924 07:18:51.594422 11525 raft_consensus.cc:2229] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:51.594468 11525 raft_consensus.cc:2258] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 07:18:51.594547 10757 catalog_manager.cc:4584] TS 7be36e4202744663900a72eb10a5ae86 (127.3.246.65:41413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2668615952424cd2b9f8c26d0056d7b4: Network error: Client connection negotiation failed: client connection to 127.3.246.65:41413: connect: Connection refused (error 111)
I20260924 07:18:51.596395 11525 ts_tablet_manager.cc:1905] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:51.597868 11525 ts_tablet_manager.cc:1918] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:51.597942 11525 log.cc:1196] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/2668615952424cd2b9f8c26d0056d7b4
I20260924 07:18:51.598162 11525 ts_tablet_manager.cc:1939] T 2668615952424cd2b9f8c26d0056d7b4 P 16eef29dfa904346bcb8ff1485075d58: Deleting consensus metadata
I20260924 07:18:51.598489 10756 catalog_manager.cc:4841] TS 16eef29dfa904346bcb8ff1485075d58 (127.3.246.67:33321): tablet 2668615952424cd2b9f8c26d0056d7b4 (table test-table [id=0982af4e9c5f42ce92579624ebd51049]) successfully deleted
I20260924 07:18:51.599953 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.599822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:34417 (local address 127.0.0.1:55484)
0924 07:18:51.599885 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.599916 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.66:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:18:51.599962 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 11282
I20260924 07:18:51.605178 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 10631
I20260924 07:18:51.610150 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 10825
2026-09-24T07:18:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4498 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:51 dist-test-slave-rc1w krb5kdc[11530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:51 dist-test-slave-rc1w krb5kdc[11530](info): set up 2 sockets
Sep 24 07:18:51 dist-test-slave-rc1w krb5kdc[11530](info): commencing operation
krb5kdc: starting...
W20260924 07:18:52.654250 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:52 dist-test-slave-rc1w krb5kdc[11530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.126.keytab.
Entry for principal oryx/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.126.keytab.
I20260924 07:18:52.776882 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37141
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.126.keytab
--principal=oryx/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:37141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:52.860791 11543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:52.861047 11543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:52.861088 11543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:52.862260 11543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:52.862336 11543 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:18:52.862365 11543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:52.862390 11543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:52.862442 11543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:52.862473 11543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:52.864503 11543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:37141
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.126.keytab
--principal=oryx/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:52.864797 11543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:52.865051 11543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:52.870082 11551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:52.870090 11548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:52.870440 11543 server_base.cc:1034] running on GCE node
W20260924 07:18:52.870100 11549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:52.870743 11543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:52.871037 11543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:52.872206 11543 hybrid_clock.cc:648] HybridClock initialized: now 1790234332872198 us; error 43 us; skew 500 ppm
Sep 24 07:18:52 dist-test-slave-rc1w krb5kdc[11530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234332, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:52.874668 11543 init.cc:377] Logged in from keytab as oryx/127.3.246.126@KRBTEST.COM (short username oryx)
I20260924 07:18:52.875211 11543 webserver.cc:460] Webserver started at http://127.3.246.126:44767/ using document root <none> and password file <none>
I20260924 07:18:52.875525 11543 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:52.875589 11543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:52.875731 11543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:52.876798 11543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "1cbef49b7b1d4b6391b3cdae6bb0118f"
format_stamp: "Formatted at 2026-09-24 07:18:52 on dist-test-slave-rc1w"
server_key: "431c4ba4aec3928dbfa81041283b686a"
server_key_iv: "a383e675c09768880a302948ba941602"
server_key_version: "encryptionkey@0"
I20260924 07:18:52.877158 11543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "1cbef49b7b1d4b6391b3cdae6bb0118f"
format_stamp: "Formatted at 2026-09-24 07:18:52 on dist-test-slave-rc1w"
server_key: "431c4ba4aec3928dbfa81041283b686a"
server_key_iv: "a383e675c09768880a302948ba941602"
server_key_version: "encryptionkey@0"
I20260924 07:18:52.878574 11543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:52.879295 11558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:52.879477 11543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:52.879567 11543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "1cbef49b7b1d4b6391b3cdae6bb0118f"
format_stamp: "Formatted at 2026-09-24 07:18:52 on dist-test-slave-rc1w"
server_key: "431c4ba4aec3928dbfa81041283b686a"
server_key_iv: "a383e675c09768880a302948ba941602"
server_key_version: "encryptionkey@0"
I20260924 07:18:52.879647 11543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:52.906632 11543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:52.909634 11543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:52.909853 11543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:52.914402 11543 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:37141
I20260924 07:18:52.914448 11620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:37141 every 8 connection(s)
I20260924 07:18:52.914784 11543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:52.915405 11621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:52.917729 11621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f: Bootstrap starting.
I20260924 07:18:52.918349 11621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:52.918639 11621 log.cc:824] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:52.919363 11621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f: No bootstrap required, opened a new log
I20260924 07:18:52.921474 11621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbef49b7b1d4b6391b3cdae6bb0118f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37141 } }
I20260924 07:18:52.921653 11621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:52.921739 11621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbef49b7b1d4b6391b3cdae6bb0118f, State: Initialized, Role: FOLLOWER
I20260924 07:18:52.921943 11621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbef49b7b1d4b6391b3cdae6bb0118f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37141 } }
I20260924 07:18:52.922003 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 11543
I20260924 07:18:52.922044 11621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:52.922086 11621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:52.922108 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:52.922139 11621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:52.922335 4057 external_mini_cluster.cc:1428] Setting key 6936618e84e9b8a795823a6b02114240
I20260924 07:18:52.922792 11621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbef49b7b1d4b6391b3cdae6bb0118f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37141 } }
I20260924 07:18:52.922899 11621 leader_election.cc:302] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cbef49b7b1d4b6391b3cdae6bb0118f; no voters:
I20260924 07:18:52.923035 11621 leader_election.cc:288] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:52.923097 11626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:52.923252 11626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [term 1 LEADER]: Becoming Leader. State: Replica: 1cbef49b7b1d4b6391b3cdae6bb0118f, State: Running, Role: LEADER
I20260924 07:18:52.923278 11621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:52.923339 11626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbef49b7b1d4b6391b3cdae6bb0118f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37141 } }
I20260924 07:18:52.923764 11626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cbef49b7b1d4b6391b3cdae6bb0118f. Latest consensus state: current_term: 1 leader_uuid: "1cbef49b7b1d4b6391b3cdae6bb0118f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbef49b7b1d4b6391b3cdae6bb0118f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37141 } } }
I20260924 07:18:52.923833 11626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:52.924721 11627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cbef49b7b1d4b6391b3cdae6bb0118f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbef49b7b1d4b6391b3cdae6bb0118f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37141 } } }
I20260924 07:18:52.925025 11627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f [sys.catalog]: This master's current role is: LEADER
W20260924 07:18:52.925282 11641 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:18:52.925338 11641 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:18:52.927376 11632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:52.927561 11632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:52.929960 11632 catalog_manager.cc:1347] Generated new cluster ID: 914a7fbcdf634d3aa42cae70b04f3815
I20260924 07:18:52.930007 11632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:18:52 dist-test-slave-rc1w krb5kdc[11530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:52.935618 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.922857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60034 (local address 127.3.246.126:37141)
0924 07:18:52.923046 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:52.923058 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:52.923095 (+ 37us) server_negotiation.cc:407] Connection header received
0924 07:18:52.923222 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:52.923229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:52.923292 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:52.924335 (+ 1043us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:52.924438 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.925944 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.926913 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.927747 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.927859 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.931071 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:52.931076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:52.931076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:52.931089 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:52.933297 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:52.933794 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:52.933797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:52.933797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:52.933938 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:52.934165 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:52.934165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:52.934166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:52.934517 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:52.935149 (+ 632us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:52.935228 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 07:18:52.935408 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260924 07:18:52.944060 11632 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:52.944770 11632 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:52.952953 11632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f: Generated new TSK 0
I20260924 07:18:52.953135 11632 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab.
Entry for principal oryx/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab.
I20260924 07:18:52.982753 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37141
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab
--principal=oryx/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:53.063972 11647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:53.064177 11647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:53.064199 11647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:53.065352 11647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:53.065392 11647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:53.065412 11647 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:18:53.065430 11647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:53.067166 11647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab
--principal=oryx/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.246.126:37141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:53.067406 11647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:53.067646 11647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:53.072734 11653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:53.072743 11652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:53.072786 11647 server_base.cc:1034] running on GCE node
W20260924 07:18:53.072743 11655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:53.073194 11647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:53.073453 11647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:53.074599 11647 hybrid_clock.cc:648] HybridClock initialized: now 1790234333074587 us; error 34 us; skew 500 ppm
Sep 24 07:18:53 dist-test-slave-rc1w krb5kdc[11530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234333, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:53.077193 11647 init.cc:377] Logged in from keytab as oryx/127.3.246.65@KRBTEST.COM (short username oryx)
I20260924 07:18:53.077629 11647 webserver.cc:460] Webserver started at http://127.3.246.65:37061/ using document root <none> and password file <none>
I20260924 07:18:53.077906 11647 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:53.077981 11647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:53.078140 11647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:53.079180 11647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec"
format_stamp: "Formatted at 2026-09-24 07:18:53 on dist-test-slave-rc1w"
server_key: "9b242b8914a35173730093c916b6a651"
server_key_iv: "5c1f7241a6695f96ec912d91032d6480"
server_key_version: "encryptionkey@0"
I20260924 07:18:53.079591 11647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec"
format_stamp: "Formatted at 2026-09-24 07:18:53 on dist-test-slave-rc1w"
server_key: "9b242b8914a35173730093c916b6a651"
server_key_iv: "5c1f7241a6695f96ec912d91032d6480"
server_key_version: "encryptionkey@0"
I20260924 07:18:53.080919 11647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:18:53.081818 11662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.082006 11647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:53.082106 11647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec"
format_stamp: "Formatted at 2026-09-24 07:18:53 on dist-test-slave-rc1w"
server_key: "9b242b8914a35173730093c916b6a651"
server_key_iv: "5c1f7241a6695f96ec912d91032d6480"
server_key_version: "encryptionkey@0"
I20260924 07:18:53.082195 11647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:53.102331 11647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:53.104179 11647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:53.104331 11647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:53.104867 11647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:53.104907 11647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.104959 11647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:53.104984 11647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.112133 11647 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:39661
I20260924 07:18:53.112200 11777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:39661 every 8 connection(s)
Sep 24 07:18:53 dist-test-slave-rc1w krb5kdc[11530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234333, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:53.112562 11647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:53.115731 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.105413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:42749 (local address 127.3.246.126:37141)
0924 07:18:53.105488 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.105491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.105721 (+ 230us) server_negotiation.cc:407] Connection header received
0924 07:18:53.106310 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.106312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.106331 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.106408 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:53.107286 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.108138 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.109791 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.110124 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.110211 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.112954 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:53.112955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:53.112956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:53.112968 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:53.114293 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.114712 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.114713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.114714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.114780 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.115093 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.115094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.115094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.115223 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:53.115282 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.115574 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.115660 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:18:53.115824 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.105364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37141 (local address 127.3.246.65:42749)
0924 07:18:53.105554 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.105596 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.106180 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.106507 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.106510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.106881 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:53.107164 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.107170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.108284 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.108285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.109675 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.109679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.110215 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.110216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.110329 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.110912 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:53.110923 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:53.112822 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:53.114372 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.114378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.114380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.114600 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.114902 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.114903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.114904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.115011 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.115305 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:53.115311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:53.115445 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.115535 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.115658 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260924 07:18:53.116384 11778 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37141
I20260924 07:18:53.116494 11778 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:53.116669 11778 heartbeater.cc:511] Master 127.3.246.126:37141 requested a full tablet report, sending...
I20260924 07:18:53.117275 11574 ts_manager.cc:194] Registered new tserver with Master: a3fd817383dd477f9ecb3d7a1b30f9ec (127.3.246.65:39661)
I20260924 07:18:53.117569 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 11647
I20260924 07:18:53.117705 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:53.117964 4057 external_mini_cluster.cc:1428] Setting key b10e01a33e897b59592ab9e33c9c8c7b
I20260924 07:18:53.118317 11574 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:42749
I20260924 07:18:53.124505 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.117490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37141 (local address 127.3.246.65:37791)
0924 07:18:53.117631 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.117643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.117750 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.117976 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.117977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.118170 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:53.118230 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.118232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.119115 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.119116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.119996 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.119999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.120531 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.120532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.120578 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.121034 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:53.121038 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:53.121447 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:53.123207 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.123208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.123209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.123491 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.123704 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.123705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.123706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.123791 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.124083 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:53.124084 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:53.124141 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.124230 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.124349 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260924 07:18:53.124519 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.117484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:37791 (local address 127.3.246.126:37141)
0924 07:18:53.117524 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.117526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.117723 (+ 197us) server_negotiation.cc:407] Connection header received
0924 07:18:53.117878 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.117880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.117894 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.117960 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:53.118294 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.118998 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.120107 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.120443 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.120530 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.121535 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:53.121536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:53.121536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:53.121542 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:53.123111 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.123571 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.123572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.123572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.123631 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.123894 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.123895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.123895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.124009 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:53.124066 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.124248 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.124319 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.3.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab.
Entry for principal oryx/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab.
I20260924 07:18:53.145488 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37141
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab
--principal=oryx/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:53.226378 11787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:53.226611 11787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:53.226649 11787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:53.227810 11787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:53.227880 11787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:53.227913 11787 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:18:53.227952 11787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:53.229426 11787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab
--principal=oryx/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.246.126:37141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:53.229748 11787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:53.229983 11787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:53.234865 11793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:53.234939 11795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:53.234867 11792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:53.235121 11787 server_base.cc:1034] running on GCE node
I20260924 07:18:53.235385 11787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:53.235641 11787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:53.236809 11787 hybrid_clock.cc:648] HybridClock initialized: now 1790234333236804 us; error 36 us; skew 500 ppm
Sep 24 07:18:53 dist-test-slave-rc1w krb5kdc[11530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234333, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:53.239387 11787 init.cc:377] Logged in from keytab as oryx/127.3.246.66@KRBTEST.COM (short username oryx)
I20260924 07:18:53.239894 11787 webserver.cc:460] Webserver started at http://127.3.246.66:36165/ using document root <none> and password file <none>
I20260924 07:18:53.240139 11787 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:53.240207 11787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:53.240346 11787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:53.241361 11787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa"
format_stamp: "Formatted at 2026-09-24 07:18:53 on dist-test-slave-rc1w"
server_key: "9f1645f3f6507927b9083c4ebcc77ae6"
server_key_iv: "1e8f6d3f2bbff9957763dd13a3d80aa7"
server_key_version: "encryptionkey@0"
I20260924 07:18:53.241814 11787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa"
format_stamp: "Formatted at 2026-09-24 07:18:53 on dist-test-slave-rc1w"
server_key: "9f1645f3f6507927b9083c4ebcc77ae6"
server_key_iv: "1e8f6d3f2bbff9957763dd13a3d80aa7"
server_key_version: "encryptionkey@0"
I20260924 07:18:53.243131 11787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:53.243896 11802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.244078 11787 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.244175 11787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa"
format_stamp: "Formatted at 2026-09-24 07:18:53 on dist-test-slave-rc1w"
server_key: "9f1645f3f6507927b9083c4ebcc77ae6"
server_key_iv: "1e8f6d3f2bbff9957763dd13a3d80aa7"
server_key_version: "encryptionkey@0"
I20260924 07:18:53.244263 11787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:53.262882 11787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:53.265254 11787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:53.265432 11787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:53.266024 11787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:53.266085 11787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.266129 11787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:53.266157 11787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:18:53 dist-test-slave-rc1w krb5kdc[11530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234333, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:53.273571 11787 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:41327
I20260924 07:18:53.273603 11917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:41327 every 8 connection(s)
I20260924 07:18:53.274049 11787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:53.276551 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.266489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:55679 (local address 127.3.246.126:37141)
0924 07:18:53.266563 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.266565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.266822 (+ 257us) server_negotiation.cc:407] Connection header received
0924 07:18:53.267583 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.267585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.267596 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.267663 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:53.268613 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.269266 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.271031 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.271309 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.271394 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.273960 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:53.273962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:53.273962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:53.273970 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:53.275445 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.275792 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.275793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.275793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.275845 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.276050 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.276051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.276051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.276144 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:53.276187 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.276426 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.276489 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:18:53.276773 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.266461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37141 (local address 127.3.246.66:55679)
0924 07:18:53.266678 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.266733 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.267461 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.267747 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.267752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.268174 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:53.268498 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.268505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.269373 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.269375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.270926 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.270931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.271395 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.271396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.271559 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.272043 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:53.272055 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:53.273865 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:53.275522 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.275528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.275530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.275737 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.275905 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.275906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.275906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.276002 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.276194 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:53.276198 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:53.276305 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.276398 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.276517 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":109,"threads_started":1}
I20260924 07:18:53.277366 11918 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37141
I20260924 07:18:53.277448 11918 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:53.277626 11918 heartbeater.cc:511] Master 127.3.246.126:37141 requested a full tablet report, sending...
I20260924 07:18:53.278050 11575 ts_manager.cc:194] Registered new tserver with Master: e5e08ed215ad4a76a16a3816f9a7ddfa (127.3.246.66:41327)
I20260924 07:18:53.278559 11575 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:55679
I20260924 07:18:53.280479 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 11787
I20260924 07:18:53.280607 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:53.280867 4057 external_mini_cluster.cc:1428] Setting key b53c6fd9dc7a530d9322166496ed50cc
I20260924 07:18:53.284484 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.278525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:37321 (local address 127.3.246.126:37141)
0924 07:18:53.278565 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.278566 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.278708 (+ 142us) server_negotiation.cc:407] Connection header received
0924 07:18:53.278733 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.278734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.278745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.278795 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:53.279185 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.279646 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.280291 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.280568 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.280629 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.281750 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:53.281751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:53.281752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:53.281759 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:53.283372 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.283776 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.283777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.283777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.283824 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.284024 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.284024 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.284025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.284149 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:53.284200 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.284372 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.284428 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:18:53.284651 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.278431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37141 (local address 127.3.246.66:37321)
0924 07:18:53.278572 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.278583 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.278648 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.278803 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.278803 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.279015 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:53.279083 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.279085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.279741 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.279742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.280218 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.280219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.280653 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.280654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.280722 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.281235 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:53.281240 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:53.281635 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:53.283453 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.283454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.283455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.283693 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.283891 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.283892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.283892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.283962 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.284218 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:53.284219 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:53.284271 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.284345 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.284455 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for oryx/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab.
Entry for principal oryx/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab.
I20260924 07:18:53.309455 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37141
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab
--principal=oryx/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:53.391748 11927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:53.391958 11927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:53.391983 11927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:53.393157 11927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:53.393198 11927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:53.393218 11927 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:18:53.393235 11927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:53.394790 11927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab
--principal=oryx/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.246.126:37141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:53.395025 11927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:53.395257 11927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:53.400508 11932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:53.400523 11935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:53.400573 11927 server_base.cc:1034] running on GCE node
W20260924 07:18:53.400523 11933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:53.400916 11927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:53.401120 11927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:53.402259 11927 hybrid_clock.cc:648] HybridClock initialized: now 1790234333402233 us; error 36 us; skew 500 ppm
Sep 24 07:18:53 dist-test-slave-rc1w krb5kdc[11530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234333, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:53.404773 11927 init.cc:377] Logged in from keytab as oryx/127.3.246.67@KRBTEST.COM (short username oryx)
I20260924 07:18:53.405256 11927 webserver.cc:460] Webserver started at http://127.3.246.67:36315/ using document root <none> and password file <none>
I20260924 07:18:53.405498 11927 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:53.405542 11927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:53.405736 11927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:53.406800 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "cff0c1f3b43742a99bec47debb039680"
format_stamp: "Formatted at 2026-09-24 07:18:53 on dist-test-slave-rc1w"
server_key: "252f84ce6e76d564dc61cd3c039ea2d0"
server_key_iv: "1d7141e793b70a9c635b6a3e40221eff"
server_key_version: "encryptionkey@0"
I20260924 07:18:53.407153 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "cff0c1f3b43742a99bec47debb039680"
format_stamp: "Formatted at 2026-09-24 07:18:53 on dist-test-slave-rc1w"
server_key: "252f84ce6e76d564dc61cd3c039ea2d0"
server_key_iv: "1d7141e793b70a9c635b6a3e40221eff"
server_key_version: "encryptionkey@0"
I20260924 07:18:53.408679 11927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:18:53.409433 11942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.409606 11927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:18:53.409734 11927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "cff0c1f3b43742a99bec47debb039680"
format_stamp: "Formatted at 2026-09-24 07:18:53 on dist-test-slave-rc1w"
server_key: "252f84ce6e76d564dc61cd3c039ea2d0"
server_key_iv: "1d7141e793b70a9c635b6a3e40221eff"
server_key_version: "encryptionkey@0"
I20260924 07:18:53.409832 11927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:53.417775 11927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:53.419778 11927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:53.419940 11927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:53.420430 11927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:53.420497 11927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.420539 11927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:53.420567 11927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:18:53 dist-test-slave-rc1w krb5kdc[11530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234333, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:53.427979 11927 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:40629
I20260924 07:18:53.428028 12057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:40629 every 8 connection(s)
I20260924 07:18:53.428395 11927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:53.431218 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.420972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:48905 (local address 127.3.246.126:37141)
0924 07:18:53.421045 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.421048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.421277 (+ 229us) server_negotiation.cc:407] Connection header received
0924 07:18:53.422040 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.422042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.422053 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.422171 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:53.423093 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.423889 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.425575 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.425914 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.425993 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.428284 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:53.428286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:53.428286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:53.428295 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:53.429989 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.430367 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.430368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.430368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.430425 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.430645 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.430646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.430646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.430771 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:53.430868 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.431083 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.431148 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:18:53.431362 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.420946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37141 (local address 127.3.246.67:48905)
0924 07:18:53.421127 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.421176 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.421934 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.422232 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.422237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.422642 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:53.422937 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.422945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.423982 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.423983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.425476 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.425479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.426004 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.426006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.426132 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.426693 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:53.426702 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:53.428171 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:53.430062 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.430068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.430070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.430286 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.430490 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.430491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.430491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.430582 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.430833 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:53.430837 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:53.430963 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.431047 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.431162 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260924 07:18:53.431804 12058 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37141
I20260924 07:18:53.431885 12058 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:53.432077 12058 heartbeater.cc:511] Master 127.3.246.126:37141 requested a full tablet report, sending...
I20260924 07:18:53.432446 11574 ts_manager.cc:194] Registered new tserver with Master: cff0c1f3b43742a99bec47debb039680 (127.3.246.67:40629)
I20260924 07:18:53.432888 11574 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:48905
I20260924 07:18:53.434283 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 11927
I20260924 07:18:53.434409 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:53.434620 4057 external_mini_cluster.cc:1428] Setting key 0f05aee4445cff4ef64be71629b488fa
I20260924 07:18:53.435487 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:18:53.439107 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.432950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:49925 (local address 127.3.246.126:37141)
0924 07:18:53.432991 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.432993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.433096 (+ 103us) server_negotiation.cc:407] Connection header received
0924 07:18:53.433150 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.433150 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.433169 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.433225 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:53.433566 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.434127 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.435040 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.435308 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.435428 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.436376 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:53.436378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:53.436378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:53.436385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:53.437948 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.438317 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.438319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.438319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.438377 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.438588 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.438589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.438589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.438696 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:53.438788 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.438945 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.439018 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:18:53.439123 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.432852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37141 (local address 127.3.246.67:49925)
0924 07:18:53.433034 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.433045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.433104 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.433233 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.433234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.433423 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:53.433492 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.433494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.434248 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.434254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.434960 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.434962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.435431 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.435432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.435501 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.435955 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:53.435959 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:53.436286 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:53.438047 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.438048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.438049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.438218 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.438456 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.438457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.438457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.438521 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.438784 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:53.438785 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:53.438830 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.438919 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.439021 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
Sep 24 07:18:53 dist-test-slave-rc1w krb5kdc[11530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234333, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:18:53 dist-test-slave-rc1w krb5kdc[11530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:53 dist-test-slave-rc1w krb5kdc[11530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:18:53.457511 11625 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:53.451992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60036 (local address 127.3.246.126:37141)
0924 07:18:53.452063 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.452064 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.452073 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:53.452096 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.452097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.452105 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.452155 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:53.452758 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.453349 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.454180 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.454378 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.454490 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.457389 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60036: BlockingRecv error: recv got EOF from 127.0.0.1:60036 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 07:18:53 dist-test-slave-rc1w krb5kdc[11530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234333, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:53.464537 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.458262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.3.246.126:37141)
0924 07:18:53.458333 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.458335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.458344 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:53.458408 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.458408 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.458416 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.458465 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:53.458895 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.459343 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.459976 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.460141 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.460187 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.462270 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:53.462271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:53.462272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:53.462278 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:53.463640 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.463952 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.463953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.463953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.464002 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.464166 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.464167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.464167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.464262 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:53.464339 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.464419 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.464473 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:18:53.466392 11574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:53.467978 11574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:53.474571 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.470407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:40629 (local address 127.0.0.1:34630)
0924 07:18:53.470800 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.470809 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.470854 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.471156 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.471157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.471159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:53.471214 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.471216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.472127 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.472128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.473288 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.473291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.474255 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.474256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.474333 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.474346 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.474429 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.474487 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":87,"threads_started":1}
I20260924 07:18:53.474611 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.470523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.3.246.67:40629)
0924 07:18:53.470698 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.470702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.470931 (+ 229us) server_negotiation.cc:407] Connection header received
0924 07:18:53.470983 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.470986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.471004 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.471071 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:53.471294 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.472036 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.473377 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.474159 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.474238 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.474329 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.474447 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.474487 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260924 07:18:53.474757 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.470504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.3.246.65:39661)
0924 07:18:53.470661 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.470667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.470824 (+ 157us) server_negotiation.cc:407] Connection header received
0924 07:18:53.470918 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.470921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.470938 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.471029 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:53.471202 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.472374 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.473575 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.474275 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.474315 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.474382 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.474598 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.474645 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260924 07:18:53.474777 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.470848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.3.246.66:41327)
0924 07:18:53.471149 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.471154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.471164 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:18:53.471200 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.471202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.471216 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.471273 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:53.471467 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.472504 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.473665 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.474228 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.474299 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.474373 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.474611 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.474673 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":81,"threads_started":1}
I20260924 07:18:53.475008 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.470370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:39661 (local address 127.0.0.1:35542)
0924 07:18:53.470701 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.470740 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.470826 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.471034 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.471037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.471040 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:53.471108 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.471111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.472473 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.472474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.473481 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.473484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.474444 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.474444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.474504 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.474507 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.474889 (+ 382us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.474942 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":94,"threads_started":1}
I20260924 07:18:53.475169 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.470603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:41327 (local address 127.0.0.1:34530)
0924 07:18:53.470939 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.470949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.470983 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.471358 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.471359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.471361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:53.471392 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.471393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.472592 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.472593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.473565 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.473567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.474406 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.474407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.474460 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.474463 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.474508 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.474539 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":63,"threads_started":1}
I20260924 07:18:53.475622 11991 tablet_service.cc:1467] Processing CreateTablet for tablet dba605a010194290bfb5f9359c6708e5 (DEFAULT_TABLE table=test-table [id=fa427dc57e254218a701fe36fd825243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:53.475683 11712 tablet_service.cc:1467] Processing CreateTablet for tablet dba605a010194290bfb5f9359c6708e5 (DEFAULT_TABLE table=test-table [id=fa427dc57e254218a701fe36fd825243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:53.475752 11852 tablet_service.cc:1467] Processing CreateTablet for tablet dba605a010194290bfb5f9359c6708e5 (DEFAULT_TABLE table=test-table [id=fa427dc57e254218a701fe36fd825243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:53.476009 11712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba605a010194290bfb5f9359c6708e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:53.476022 11991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba605a010194290bfb5f9359c6708e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:53.476043 11852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba605a010194290bfb5f9359c6708e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:53.479341 12086 tablet_bootstrap.cc:492] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa: Bootstrap starting.
I20260924 07:18:53.479710 12087 tablet_bootstrap.cc:492] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680: Bootstrap starting.
I20260924 07:18:53.480000 12086 tablet_bootstrap.cc:654] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:53.480278 12087 tablet_bootstrap.cc:654] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:53.480335 12086 log.cc:824] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:53.480583 12088 tablet_bootstrap.cc:492] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec: Bootstrap starting.
I20260924 07:18:53.480615 12087 log.cc:824] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:53.481216 12088 tablet_bootstrap.cc:654] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:53.481355 12087 tablet_bootstrap.cc:492] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680: No bootstrap required, opened a new log
I20260924 07:18:53.481427 12087 ts_tablet_manager.cc:1397] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:53.481482 12086 tablet_bootstrap.cc:492] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa: No bootstrap required, opened a new log
I20260924 07:18:53.481549 12086 ts_tablet_manager.cc:1397] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:53.481582 12088 log.cc:824] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:53.482429 12088 tablet_bootstrap.cc:492] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec: No bootstrap required, opened a new log
I20260924 07:18:53.482519 12088 ts_tablet_manager.cc:1397] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:53.483461 12087 raft_consensus.cc:348] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } }
I20260924 07:18:53.483461 12086 raft_consensus.cc:348] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } }
I20260924 07:18:53.483631 12087 raft_consensus.cc:374] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:53.483631 12086 raft_consensus.cc:374] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:53.483673 12087 raft_consensus.cc:729] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff0c1f3b43742a99bec47debb039680, State: Initialized, Role: FOLLOWER
I20260924 07:18:53.483677 12086 raft_consensus.cc:729] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e08ed215ad4a76a16a3816f9a7ddfa, State: Initialized, Role: FOLLOWER
I20260924 07:18:53.483791 12087 consensus_queue.cc:260] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } }
I20260924 07:18:53.483805 12086 consensus_queue.cc:260] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } }
I20260924 07:18:53.484066 12087 ts_tablet_manager.cc:1428] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 07:18:53.484066 12086 ts_tablet_manager.cc:1428] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 07:18:53.484186 12058 heartbeater.cc:503] Master 127.3.246.126:37141 was elected leader, sending a full tablet report...
I20260924 07:18:53.484249 11918 heartbeater.cc:503] Master 127.3.246.126:37141 was elected leader, sending a full tablet report...
I20260924 07:18:53.484644 12088 raft_consensus.cc:348] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } }
I20260924 07:18:53.484804 12088 raft_consensus.cc:374] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:53.484851 12088 raft_consensus.cc:729] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3fd817383dd477f9ecb3d7a1b30f9ec, State: Initialized, Role: FOLLOWER
I20260924 07:18:53.484957 12088 consensus_queue.cc:260] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } }
I20260924 07:18:53.485250 11778 heartbeater.cc:503] Master 127.3.246.126:37141 was elected leader, sending a full tablet report...
I20260924 07:18:53.485260 12088 ts_tablet_manager.cc:1428] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260924 07:18:53.525022 11919 tablet.cc:2369] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:18:53.609881 12094 raft_consensus.cc:482] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:53.610165 12094 raft_consensus.cc:504] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } }
I20260924 07:18:53.610548 12094 leader_election.cc:288] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cff0c1f3b43742a99bec47debb039680 (127.3.246.67:40629), e5e08ed215ad4a76a16a3816f9a7ddfa (127.3.246.66:41327)
W20260924 07:18:53.613338 11779 tablet.cc:2369] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:18:53.614730 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.610988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:35193 (local address 127.3.246.66:41327)
0924 07:18:53.611059 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.611061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.611091 (+ 30us) server_negotiation.cc:407] Connection header received
0924 07:18:53.611118 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.611119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.611128 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.611183 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:53.611367 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.612136 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.613772 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.614338 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.614414 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.614423 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.614623 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.614664 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:18:53.614768 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.610899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:49705 (local address 127.3.246.67:40629)
0924 07:18:53.610956 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.610958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.610970 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:18:53.611023 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.611024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.611035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.611105 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:53.611366 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.612173 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.613766 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.614333 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.614414 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.614423 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.614623 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.614661 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:18:53.614753 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.610877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:41327 (local address 127.3.246.65:35193)
0924 07:18:53.611002 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.611012 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.611051 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.611199 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.611200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.611202 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:53.611261 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.611262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.612272 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.612273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.613640 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.613644 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.614440 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.614441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.614533 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.614539 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.614602 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.614671 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":20,"thread_start_us":67,"threads_started":1}
I20260924 07:18:53.614765 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.610759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:40629 (local address 127.3.246.65:49705)
0924 07:18:53.610822 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.610836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.610910 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.611123 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.611124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.611126 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:53.611236 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.611238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.612272 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.612273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.613640 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.613644 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.614440 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.614441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.614533 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.614538 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.614602 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.614650 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":95}
I20260924 07:18:53.615272 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba605a010194290bfb5f9359c6708e5" candidate_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" is_pre_election: true
I20260924 07:18:53.615276 12010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba605a010194290bfb5f9359c6708e5" candidate_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff0c1f3b43742a99bec47debb039680" is_pre_election: true
I20260924 07:18:53.615451 11872 raft_consensus.cc:2454] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3fd817383dd477f9ecb3d7a1b30f9ec in term 0.
I20260924 07:18:53.615468 12010 raft_consensus.cc:2454] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3fd817383dd477f9ecb3d7a1b30f9ec in term 0.
I20260924 07:18:53.615725 11665 leader_election.cc:302] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3fd817383dd477f9ecb3d7a1b30f9ec, cff0c1f3b43742a99bec47debb039680; no voters:
I20260924 07:18:53.615850 12094 raft_consensus.cc:2781] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:53.615938 12094 raft_consensus.cc:482] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:53.615975 12094 raft_consensus.cc:3037] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:53.617027 12094 raft_consensus.cc:504] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } }
I20260924 07:18:53.617168 12094 leader_election.cc:288] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [CANDIDATE]: Term 1 election: Requested vote from peers cff0c1f3b43742a99bec47debb039680 (127.3.246.67:40629), e5e08ed215ad4a76a16a3816f9a7ddfa (127.3.246.66:41327)
I20260924 07:18:53.617393 12010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba605a010194290bfb5f9359c6708e5" candidate_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff0c1f3b43742a99bec47debb039680"
I20260924 07:18:53.617398 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba605a010194290bfb5f9359c6708e5" candidate_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa"
I20260924 07:18:53.617482 12010 raft_consensus.cc:3037] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:53.617498 11872 raft_consensus.cc:3037] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:53.618175 11872 raft_consensus.cc:2454] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3fd817383dd477f9ecb3d7a1b30f9ec in term 1.
I20260924 07:18:53.618405 11665 leader_election.cc:302] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3fd817383dd477f9ecb3d7a1b30f9ec, e5e08ed215ad4a76a16a3816f9a7ddfa; no voters:
I20260924 07:18:53.618520 12010 raft_consensus.cc:2454] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3fd817383dd477f9ecb3d7a1b30f9ec in term 1.
I20260924 07:18:53.618568 12094 raft_consensus.cc:2781] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:53.618762 12094 raft_consensus.cc:686] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 LEADER]: Becoming Leader. State: Replica: a3fd817383dd477f9ecb3d7a1b30f9ec, State: Running, Role: LEADER
I20260924 07:18:53.618860 12094 consensus_queue.cc:237] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } }
I20260924 07:18:53.619761 11575 catalog_manager.cc:5495] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec reported cstate change: term changed from 0 to 1, leader changed from <none> to a3fd817383dd477f9ecb3d7a1b30f9ec (127.3.246.65). New cstate: current_term: 1 leader_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } health_report { overall_health: UNKNOWN } } }
I20260924 07:18:53.659647 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.657206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.3.246.65:39661)
0924 07:18:53.657282 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.657284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.657292 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:18:53.657329 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.657330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.657347 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.657449 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:53.657590 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.658118 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.658919 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.659110 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.659179 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.659320 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:53.659412 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:53.659470 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.659507 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.659526 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:18:53.661440 12010 raft_consensus.cc:1261] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Refusing update from remote peer a3fd817383dd477f9ecb3d7a1b30f9ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:53.661444 11872 raft_consensus.cc:1261] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Refusing update from remote peer a3fd817383dd477f9ecb3d7a1b30f9ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:53.661861 12094 consensus_queue.cc:1035] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:53.662010 12096 consensus_queue.cc:1035] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:53.664336 12100 mvcc.cc:205] Tried to move back new op lower bound from 7332799830674534400 to 7332799830503616512. Current Snapshot: MvccSnapshot[applied={T|T < 7332799830674534400}]
I20260924 07:18:53.664878 12101 mvcc.cc:205] Tried to move back new op lower bound from 7332799830674534400 to 7332799830503616512. Current Snapshot: MvccSnapshot[applied={T|T < 7332799830674534400}]
I20260924 07:18:53.666961 11574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:53.667048 11574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:53.667809 11574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f: Sending DeleteTablet for 3 replicas of tablet dba605a010194290bfb5f9359c6708e5
I20260924 07:18:53.668133 11991 tablet_service.cc:1514] Processing DeleteTablet for tablet dba605a010194290bfb5f9359c6708e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:53 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:34630
I20260924 07:18:53.668330 11852 tablet_service.cc:1514] Processing DeleteTablet for tablet dba605a010194290bfb5f9359c6708e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:53 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:34530
I20260924 07:18:53.668371 12109 tablet_replica.cc:330] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680: stopping tablet replica
I20260924 07:18:53.668486 12109 raft_consensus.cc:2229] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:53.668684 11712 tablet_service.cc:1514] Processing DeleteTablet for tablet dba605a010194290bfb5f9359c6708e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:53 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:35542
I20260924 07:18:53.668859 12109 raft_consensus.cc:2258] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:53.668852 11574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:53.668992 11574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:53.669291 12109 ts_tablet_manager.cc:1905] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:53.669381 12110 tablet_replica.cc:330] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa: stopping tablet replica
I20260924 07:18:53.669963 12111 tablet_replica.cc:330] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec: stopping tablet replica
I20260924 07:18:53.670115 12111 raft_consensus.cc:2229] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:53.670269 12111 raft_consensus.cc:2258] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:53.670639 11991 tablet_service.cc:1467] Processing CreateTablet for tablet a636b870ac0145c69895a371b77bda44 (DEFAULT_TABLE table=test-table [id=46573a73c541490ea85f1cbcb1c554f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:53.670732 11991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a636b870ac0145c69895a371b77bda44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:53.670732 11712 tablet_service.cc:1467] Processing CreateTablet for tablet a636b870ac0145c69895a371b77bda44 (DEFAULT_TABLE table=test-table [id=46573a73c541490ea85f1cbcb1c554f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:53.670843 11712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a636b870ac0145c69895a371b77bda44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:53.671224 12109 ts_tablet_manager.cc:1918] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:53.671284 12109 log.cc:1196] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/dba605a010194290bfb5f9359c6708e5
I20260924 07:18:53.671538 12109 ts_tablet_manager.cc:1939] T dba605a010194290bfb5f9359c6708e5 P cff0c1f3b43742a99bec47debb039680: Deleting consensus metadata
I20260924 07:18:53.671998 12087 tablet_bootstrap.cc:492] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680: Bootstrap starting.
I20260924 07:18:53.672246 11561 catalog_manager.cc:4841] TS cff0c1f3b43742a99bec47debb039680 (127.3.246.67:40629): tablet dba605a010194290bfb5f9359c6708e5 (table test-table [id=fa427dc57e254218a701fe36fd825243]) successfully deleted
I20260924 07:18:53.672398 11852 tablet_service.cc:1467] Processing CreateTablet for tablet a636b870ac0145c69895a371b77bda44 (DEFAULT_TABLE table=test-table [id=46573a73c541490ea85f1cbcb1c554f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:53.672529 11852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a636b870ac0145c69895a371b77bda44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:53.672597 12087 tablet_bootstrap.cc:654] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:53.673120 12088 tablet_bootstrap.cc:492] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec: Bootstrap starting.
I20260924 07:18:53.673188 12087 tablet_bootstrap.cc:492] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680: No bootstrap required, opened a new log
I20260924 07:18:53.673223 12087 ts_tablet_manager.cc:1397] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:53.673388 12087 raft_consensus.cc:348] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.673475 12087 raft_consensus.cc:374] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:53.673502 12087 raft_consensus.cc:729] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff0c1f3b43742a99bec47debb039680, State: Initialized, Role: FOLLOWER
I20260924 07:18:53.673514 12088 tablet_bootstrap.cc:654] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:53.673547 12087 consensus_queue.cc:260] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.673655 12087 ts_tablet_manager.cc:1428] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 07:18:53.673956 12111 ts_tablet_manager.cc:1905] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:53.674288 12088 tablet_bootstrap.cc:492] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec: No bootstrap required, opened a new log
I20260924 07:18:53.674366 12088 ts_tablet_manager.cc:1397] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:53.674531 12088 raft_consensus.cc:348] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.674618 12088 raft_consensus.cc:374] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:53.674646 12088 raft_consensus.cc:729] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3fd817383dd477f9ecb3d7a1b30f9ec, State: Initialized, Role: FOLLOWER
I20260924 07:18:53.674692 12088 consensus_queue.cc:260] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.674780 12088 ts_tablet_manager.cc:1428] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:53.675567 12086 tablet_bootstrap.cc:492] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Bootstrap starting.
I20260924 07:18:53.675642 12110 raft_consensus.cc:2229] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:53.675782 12110 raft_consensus.cc:2258] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:53.676100 12086 tablet_bootstrap.cc:654] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:53.676303 12111 ts_tablet_manager.cc:1918] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:53.676366 12111 log.cc:1196] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/wal/wals/dba605a010194290bfb5f9359c6708e5
I20260924 07:18:53.676613 12111 ts_tablet_manager.cc:1939] T dba605a010194290bfb5f9359c6708e5 P a3fd817383dd477f9ecb3d7a1b30f9ec: Deleting consensus metadata
I20260924 07:18:53.676623 12110 ts_tablet_manager.cc:1905] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:53.677037 11562 catalog_manager.cc:4841] TS a3fd817383dd477f9ecb3d7a1b30f9ec (127.3.246.65:39661): tablet dba605a010194290bfb5f9359c6708e5 (table test-table [id=fa427dc57e254218a701fe36fd825243]) successfully deleted
I20260924 07:18:53.678118 12110 ts_tablet_manager.cc:1918] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:53.678162 12086 tablet_bootstrap.cc:492] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa: No bootstrap required, opened a new log
I20260924 07:18:53.678192 12110 log.cc:1196] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/dba605a010194290bfb5f9359c6708e5
I20260924 07:18:53.678246 12086 ts_tablet_manager.cc:1397] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 07:18:53.678431 12086 raft_consensus.cc:348] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.678545 12086 raft_consensus.cc:374] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:53.678573 12086 raft_consensus.cc:729] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e08ed215ad4a76a16a3816f9a7ddfa, State: Initialized, Role: FOLLOWER
I20260924 07:18:53.678629 12086 consensus_queue.cc:260] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.678717 12086 ts_tablet_manager.cc:1428] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.678831 12110 ts_tablet_manager.cc:1939] T dba605a010194290bfb5f9359c6708e5 P e5e08ed215ad4a76a16a3816f9a7ddfa: Deleting consensus metadata
W20260924 07:18:53.678997 12059 tablet.cc:2369] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:18:53.679219 11561 catalog_manager.cc:4841] TS e5e08ed215ad4a76a16a3816f9a7ddfa (127.3.246.66:41327): tablet dba605a010194290bfb5f9359c6708e5 (table test-table [id=fa427dc57e254218a701fe36fd825243]) successfully deleted
I20260924 07:18:53.747517 12092 raft_consensus.cc:482] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:53.747632 12092 raft_consensus.cc:504] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.748049 12092 leader_election.cc:288] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3fd817383dd477f9ecb3d7a1b30f9ec (127.3.246.65:39661), cff0c1f3b43742a99bec47debb039680 (127.3.246.67:40629)
I20260924 07:18:53.751696 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.748449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:45965 (local address 127.3.246.67:40629)
0924 07:18:53.748499 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.748501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.748510 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:53.748539 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.748540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.748550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.748616 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:53.748865 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.749574 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.750905 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.751273 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.751382 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.751391 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.751571 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.751613 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:18:53.751714 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.748232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:39661 (local address 127.3.246.66:54657)
0924 07:18:53.748303 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.748315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.748360 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.748589 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.748590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.748591 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:53.748661 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.748663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.749382 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.749383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.750622 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.750625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.751420 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.751421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.751506 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.751510 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.751562 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.751610 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 07:18:53.751847 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.748247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:40629 (local address 127.3.246.66:45965)
0924 07:18:53.748391 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.748403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.748464 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.748641 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.748642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.748644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:53.748760 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.748762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.749719 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.749720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.750807 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.750809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.751383 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.751384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.751469 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.751473 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.751562 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.751615 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260924 07:18:53.752012 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.748325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:54657 (local address 127.3.246.65:39661)
0924 07:18:53.748443 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.748444 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.748453 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:53.748479 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.748480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.748491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.748538 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:53.748762 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.749282 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.750742 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.751295 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.751373 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.751384 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.751902 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.751947 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 07:18:53.752277 12010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a636b870ac0145c69895a371b77bda44" candidate_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff0c1f3b43742a99bec47debb039680" is_pre_election: true
I20260924 07:18:53.752322 11732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a636b870ac0145c69895a371b77bda44" candidate_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" is_pre_election: true
I20260924 07:18:53.752405 12010 raft_consensus.cc:2454] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e08ed215ad4a76a16a3816f9a7ddfa in term 0.
I20260924 07:18:53.752442 11732 raft_consensus.cc:2454] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e08ed215ad4a76a16a3816f9a7ddfa in term 0.
I20260924 07:18:53.752625 11805 leader_election.cc:302] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cff0c1f3b43742a99bec47debb039680, e5e08ed215ad4a76a16a3816f9a7ddfa; no voters:
I20260924 07:18:53.752898 12092 raft_consensus.cc:2781] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:53.752966 12092 raft_consensus.cc:482] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:53.752998 12092 raft_consensus.cc:3037] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:53.753857 12092 raft_consensus.cc:504] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.754024 12092 leader_election.cc:288] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [CANDIDATE]: Term 1 election: Requested vote from peers a3fd817383dd477f9ecb3d7a1b30f9ec (127.3.246.65:39661), cff0c1f3b43742a99bec47debb039680 (127.3.246.67:40629)
I20260924 07:18:53.754230 12010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a636b870ac0145c69895a371b77bda44" candidate_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff0c1f3b43742a99bec47debb039680"
I20260924 07:18:53.754313 12010 raft_consensus.cc:3037] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:53.754345 11732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a636b870ac0145c69895a371b77bda44" candidate_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec"
I20260924 07:18:53.754429 11732 raft_consensus.cc:3037] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:53.755147 12010 raft_consensus.cc:2454] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e08ed215ad4a76a16a3816f9a7ddfa in term 1.
I20260924 07:18:53.755401 11805 leader_election.cc:302] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cff0c1f3b43742a99bec47debb039680, e5e08ed215ad4a76a16a3816f9a7ddfa; no voters:
I20260924 07:18:53.755452 11732 raft_consensus.cc:2454] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e08ed215ad4a76a16a3816f9a7ddfa in term 1.
I20260924 07:18:53.755524 12092 raft_consensus.cc:2781] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:53.755688 12092 raft_consensus.cc:686] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 LEADER]: Becoming Leader. State: Replica: e5e08ed215ad4a76a16a3816f9a7ddfa, State: Running, Role: LEADER
I20260924 07:18:53.755766 12092 consensus_queue.cc:237] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.756631 11574 catalog_manager.cc:5495] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e08ed215ad4a76a16a3816f9a7ddfa (127.3.246.66). New cstate: current_term: 1 leader_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } health_report { overall_health: UNKNOWN } } }
Sep 24 07:18:53 dist-test-slave-rc1w krb5kdc[11530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234332, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.126@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:53.801813 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.794477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.3.246.126:37141)
0924 07:18:53.794628 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:53.794631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:53.794787 (+ 156us) server_negotiation.cc:407] Connection header received
0924 07:18:53.794825 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:53.794826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:53.794841 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:53.794906 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:53.795458 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.796007 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.796706 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.796885 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.796942 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.799130 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:53.799131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:53.799132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:53.799137 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:53.800370 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.800741 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.800742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.800742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.800810 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:53.801207 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:53.801207 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:53.801208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:53.801332 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:53.801391 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:53.801602 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 07:18:53.801716 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":100}
I20260924 07:18:53.801894 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.794375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37141 (local address 127.0.0.1:60044)
0924 07:18:53.794634 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.794645 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.794720 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.794947 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:53.794948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:53.795241 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:53.795306 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.795308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.796094 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.796095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.796626 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:53.796627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:53.796958 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:53.796959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:53.796999 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:53.797606 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:53.797611 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:53.799048 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:53.800457 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.800464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.800466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.800652 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.800886 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:53.800886 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:53.800887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:53.800943 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:53.801420 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:53.801426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:53.801472 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 07:18:53.801543 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:18:53.801743 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":121,"threads_started":1}
I20260924 07:18:53.804196 11574 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:60044:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:18:53.804428 11574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:53.805903 11712 tablet_service.cc:1467] Processing CreateTablet for tablet c0aea44f5a4a471c973d96804509fa4b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6151bfd1e08452ab2e4ef417046b442]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:18:53.805932 11852 tablet_service.cc:1467] Processing CreateTablet for tablet c0aea44f5a4a471c973d96804509fa4b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6151bfd1e08452ab2e4ef417046b442]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:18:53.805959 11991 tablet_service.cc:1467] Processing CreateTablet for tablet c0aea44f5a4a471c973d96804509fa4b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6151bfd1e08452ab2e4ef417046b442]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:18:53.806084 11991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0aea44f5a4a471c973d96804509fa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:53.806084 11852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0aea44f5a4a471c973d96804509fa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:53.806087 11712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0aea44f5a4a471c973d96804509fa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:53.807224 12088 tablet_bootstrap.cc:492] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec: Bootstrap starting.
I20260924 07:18:53.807554 12087 tablet_bootstrap.cc:492] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680: Bootstrap starting.
I20260924 07:18:53.807844 12086 tablet_bootstrap.cc:492] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa: Bootstrap starting.
I20260924 07:18:53.807984 12087 tablet_bootstrap.cc:654] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:53.808245 12086 tablet_bootstrap.cc:654] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:53.808401 12088 tablet_bootstrap.cc:654] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:53.808890 12086 tablet_bootstrap.cc:492] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa: No bootstrap required, opened a new log
I20260924 07:18:53.808949 12086 ts_tablet_manager.cc:1397] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:53.809149 12088 tablet_bootstrap.cc:492] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec: No bootstrap required, opened a new log
I20260924 07:18:53.809122 12086 raft_consensus.cc:348] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.809198 12086 raft_consensus.cc:374] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:53.809202 12088 ts_tablet_manager.cc:1397] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:53.809222 12086 raft_consensus.cc:729] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e08ed215ad4a76a16a3816f9a7ddfa, State: Initialized, Role: FOLLOWER
I20260924 07:18:53.809269 12086 consensus_queue.cc:260] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.809315 12088 raft_consensus.cc:348] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.809360 12086 ts_tablet_manager.cc:1428] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:53.809386 12088 raft_consensus.cc:374] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:53.809409 12088 raft_consensus.cc:729] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3fd817383dd477f9ecb3d7a1b30f9ec, State: Initialized, Role: FOLLOWER
I20260924 07:18:53.809450 12088 consensus_queue.cc:260] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.809489 12092 tablet_replica.cc:439] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } }
I20260924 07:18:53.809528 12088 ts_tablet_manager.cc:1428] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.809563 12092 tablet_replica.cc:442] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:53.809758 12096 tablet_replica.cc:439] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } }
I20260924 07:18:53.809839 12096 tablet_replica.cc:442] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:53.809976 12087 tablet_bootstrap.cc:492] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680: No bootstrap required, opened a new log
I20260924 07:18:53.810010 12087 ts_tablet_manager.cc:1397] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:18:53.810122 12087 raft_consensus.cc:348] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.810164 12087 raft_consensus.cc:374] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:53.810177 12087 raft_consensus.cc:729] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff0c1f3b43742a99bec47debb039680, State: Initialized, Role: FOLLOWER
I20260924 07:18:53.810240 12087 consensus_queue.cc:260] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:53.810343 12087 ts_tablet_manager.cc:1428] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.810400 12093 tablet_replica.cc:439] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } }
I20260924 07:18:53.810472 12093 tablet_replica.cc:442] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:54.019212 12096 raft_consensus.cc:482] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:54.019323 12096 raft_consensus.cc:504] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.019572 12096 leader_election.cc:288] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5e08ed215ad4a76a16a3816f9a7ddfa (127.3.246.66:41327), cff0c1f3b43742a99bec47debb039680 (127.3.246.67:40629)
I20260924 07:18:54.019868 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0aea44f5a4a471c973d96804509fa4b" candidate_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" is_pre_election: true
I20260924 07:18:54.019867 12010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0aea44f5a4a471c973d96804509fa4b" candidate_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff0c1f3b43742a99bec47debb039680" is_pre_election: true
I20260924 07:18:54.019981 12010 raft_consensus.cc:2454] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3fd817383dd477f9ecb3d7a1b30f9ec in term 0.
I20260924 07:18:54.019981 11872 raft_consensus.cc:2454] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3fd817383dd477f9ecb3d7a1b30f9ec in term 0.
I20260924 07:18:54.020188 11665 leader_election.cc:302] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3fd817383dd477f9ecb3d7a1b30f9ec, cff0c1f3b43742a99bec47debb039680; no voters:
I20260924 07:18:54.020293 12096 raft_consensus.cc:2781] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:54.020335 12096 raft_consensus.cc:482] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:54.020358 12096 raft_consensus.cc:3037] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:54.021096 12096 raft_consensus.cc:504] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.021239 12096 leader_election.cc:288] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [CANDIDATE]: Term 1 election: Requested vote from peers e5e08ed215ad4a76a16a3816f9a7ddfa (127.3.246.66:41327), cff0c1f3b43742a99bec47debb039680 (127.3.246.67:40629)
I20260924 07:18:54.021399 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0aea44f5a4a471c973d96804509fa4b" candidate_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa"
I20260924 07:18:54.021478 11872 raft_consensus.cc:3037] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:54.021422 12010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0aea44f5a4a471c973d96804509fa4b" candidate_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff0c1f3b43742a99bec47debb039680"
I20260924 07:18:54.021518 12010 raft_consensus.cc:3037] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:54.022202 11872 raft_consensus.cc:2454] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3fd817383dd477f9ecb3d7a1b30f9ec in term 1.
I20260924 07:18:54.022217 12010 raft_consensus.cc:2454] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3fd817383dd477f9ecb3d7a1b30f9ec in term 1.
I20260924 07:18:54.022383 11665 leader_election.cc:302] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3fd817383dd477f9ecb3d7a1b30f9ec, cff0c1f3b43742a99bec47debb039680; no voters:
I20260924 07:18:54.022521 12096 raft_consensus.cc:2781] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:54.022590 12096 raft_consensus.cc:686] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 LEADER]: Becoming Leader. State: Replica: a3fd817383dd477f9ecb3d7a1b30f9ec, State: Running, Role: LEADER
I20260924 07:18:54.022641 12096 consensus_queue.cc:237] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.023000 12107 tablet_replica.cc:439] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec: TxnStatusTablet state changed. Reason: New leader a3fd817383dd477f9ecb3d7a1b30f9ec. Latest consensus state: current_term: 1 leader_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } }
I20260924 07:18:54.023095 12107 tablet_replica.cc:442] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:18:54.023322 12128 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:18:54.023448 12128 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:18:54.023568 11575 catalog_manager.cc:5495] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec reported cstate change: term changed from 0 to 1, leader changed from <none> to a3fd817383dd477f9ecb3d7a1b30f9ec (127.3.246.65). New cstate: current_term: 1 leader_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } health_report { overall_health: UNKNOWN } } }
I20260924 07:18:54.040068 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.037617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.3.246.65:39661)
0924 07:18:54.037700 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.037702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.037711 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:54.037844 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.037845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.037856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.037927 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:54.038068 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.038556 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.039346 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.039541 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.039631 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.039806 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:54.039848 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:54.039911 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:54.039971 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 07:18:54.039993 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:18:54.040091 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.037525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:39661 (local address 127.0.0.1:35556)
0924 07:18:54.037614 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:54.037652 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:54.037740 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:54.037936 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:54.037937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:54.037938 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:18:54.037988 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.037991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.038637 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.038638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.039221 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.039222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.039632 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.039633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.039705 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.039719 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:18:54.039913 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:18:54.039922 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 07:18:54.039969 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 07:18:54.040019 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:18:54.041203 12010 raft_consensus.cc:1261] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Refusing update from remote peer a3fd817383dd477f9ecb3d7a1b30f9ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:54.041220 11872 raft_consensus.cc:1261] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Refusing update from remote peer a3fd817383dd477f9ecb3d7a1b30f9ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:54.041471 12096 consensus_queue.cc:1035] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:54.041608 12096 consensus_queue.cc:1035] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:54.042095 12092 tablet_replica.cc:439] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa: TxnStatusTablet state changed. Reason: New leader a3fd817383dd477f9ecb3d7a1b30f9ec. Latest consensus state: current_term: 1 leader_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } }
I20260924 07:18:54.042187 12092 tablet_replica.cc:442] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:54.042251 12093 tablet_replica.cc:439] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680: TxnStatusTablet state changed. Reason: New leader a3fd817383dd477f9ecb3d7a1b30f9ec. Latest consensus state: current_term: 1 leader_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } }
I20260924 07:18:54.042354 12093 tablet_replica.cc:442] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:54.042656 12107 tablet_replica.cc:439] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } }
I20260924 07:18:54.042740 12107 tablet_replica.cc:442] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:18:54.042914 12099 mvcc.cc:205] Tried to move back new op lower bound from 7332799832231120896 to 7332799832157245440. Current Snapshot: MvccSnapshot[applied={T|T < 7332799832231120896 or (T in {7332799832231120896})}]
I20260924 07:18:54.043467 12092 tablet_replica.cc:439] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } }
I20260924 07:18:54.043550 12092 tablet_replica.cc:442] T c0aea44f5a4a471c973d96804509fa4b P e5e08ed215ad4a76a16a3816f9a7ddfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:54.044432 12096 tablet_replica.cc:439] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } }
I20260924 07:18:54.044514 12096 tablet_replica.cc:442] T c0aea44f5a4a471c973d96804509fa4b P a3fd817383dd477f9ecb3d7a1b30f9ec: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:18:54.044641 12093 tablet_replica.cc:439] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } } }
I20260924 07:18:54.044713 12093 tablet_replica.cc:442] T c0aea44f5a4a471c973d96804509fa4b P cff0c1f3b43742a99bec47debb039680: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:18:54.047104 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.044337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.3.246.66:41327)
0924 07:18:54.044398 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.044400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.044408 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:18:54.044444 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.044445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.044466 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.044531 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:54.044820 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.045440 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.046256 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.046531 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.046615 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.046752 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:54.046866 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:54.046950 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:54.047009 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 07:18:54.047030 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:18:54.052557 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.049757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:39049 (local address 127.3.246.65:39661)
0924 07:18:54.049834 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.049835 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.049918 (+ 83us) server_negotiation.cc:407] Connection header received
0924 07:18:54.049989 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.049989 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.049997 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.050043 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:54.050288 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.050782 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.051908 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.052085 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.052142 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.052345 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:54.052370 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:54.052408 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:54.052491 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 07:18:54.052506 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:18:54.052790 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.049698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:39661 (local address 127.3.246.66:39049)
0924 07:18:54.049852 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:54.049864 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:54.049930 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:54.050072 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:54.050073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:54.050074 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:18:54.050148 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.050150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.050869 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.050870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.051805 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.051807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.052182 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.052184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.052250 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.052265 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:18:54.052432 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:18:54.052434 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:18:54.052489 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 07:18:54.052533 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260924 07:18:54.055392 11732 raft_consensus.cc:1261] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Refusing update from remote peer e5e08ed215ad4a76a16a3816f9a7ddfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:54.055588 12010 raft_consensus.cc:1261] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Refusing update from remote peer e5e08ed215ad4a76a16a3816f9a7ddfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:54.055721 12092 consensus_queue.cc:1035] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:54.055852 12116 consensus_queue.cc:1035] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:54.064965 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.062303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:53907 (local address 127.3.246.66:41327)
0924 07:18:54.062344 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.062346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.062464 (+ 118us) server_negotiation.cc:407] Connection header received
0924 07:18:54.062531 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.062532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.062543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.062592 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:54.062733 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.063203 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.064276 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.064483 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.064557 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.064745 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:54.064779 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:54.064821 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:54.064876 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 07:18:54.064897 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:18:54.064992 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.062223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:41327 (local address 127.3.246.65:53907)
0924 07:18:54.062385 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:54.062396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:54.062482 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:54.062593 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:54.062594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:54.062596 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:18:54.062659 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.062660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.063288 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.063289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.064192 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.064194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.064567 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.064568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.064628 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.064651 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:18:54.064832 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:18:54.064834 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:18:54.064875 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 07:18:54.064917 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260924 07:18:54.074860 11574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:54.074962 11574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:54.075776 11574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f: Sending DeleteTablet for 3 replicas of tablet a636b870ac0145c69895a371b77bda44
I20260924 07:18:54.076104 11712 tablet_service.cc:1514] Processing DeleteTablet for tablet a636b870ac0145c69895a371b77bda44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:54 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:35542
I20260924 07:18:54.076155 11991 tablet_service.cc:1514] Processing DeleteTablet for tablet a636b870ac0145c69895a371b77bda44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:54 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:34630
I20260924 07:18:54.076206 12111 tablet_replica.cc:330] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec: stopping tablet replica
I20260924 07:18:54.076254 12109 tablet_replica.cc:330] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680: stopping tablet replica
I20260924 07:18:54.076268 12111 raft_consensus.cc:2229] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:54.076298 12109 raft_consensus.cc:2229] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:54.076305 12111 raft_consensus.cc:2258] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:54.076364 12109 raft_consensus.cc:2258] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:54.076649 12111 ts_tablet_manager.cc:1905] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:54.076862 12109 ts_tablet_manager.cc:1905] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:54.078636 12111 ts_tablet_manager.cc:1918] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:54.078680 12109 ts_tablet_manager.cc:1918] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:54.078696 12111 log.cc:1196] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/wal/wals/a636b870ac0145c69895a371b77bda44
I20260924 07:18:54.078735 12109 log.cc:1196] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/a636b870ac0145c69895a371b77bda44
I20260924 07:18:54.078934 12111 ts_tablet_manager.cc:1939] T a636b870ac0145c69895a371b77bda44 P a3fd817383dd477f9ecb3d7a1b30f9ec: Deleting consensus metadata
I20260924 07:18:54.078991 12109 ts_tablet_manager.cc:1939] T a636b870ac0145c69895a371b77bda44 P cff0c1f3b43742a99bec47debb039680: Deleting consensus metadata
I20260924 07:18:54.079231 11562 catalog_manager.cc:4841] TS a3fd817383dd477f9ecb3d7a1b30f9ec (127.3.246.65:39661): tablet a636b870ac0145c69895a371b77bda44 (table test-table [id=46573a73c541490ea85f1cbcb1c554f7]) successfully deleted
I20260924 07:18:54.079299 11561 catalog_manager.cc:4841] TS cff0c1f3b43742a99bec47debb039680 (127.3.246.67:40629): tablet a636b870ac0145c69895a371b77bda44 (table test-table [id=46573a73c541490ea85f1cbcb1c554f7]) successfully deleted
I20260924 07:18:54.080523 11852 tablet_service.cc:1514] Processing DeleteTablet for tablet a636b870ac0145c69895a371b77bda44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:54 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:34530
I20260924 07:18:54.080618 12110 tablet_replica.cc:330] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa: stopping tablet replica
I20260924 07:18:54.080673 12110 raft_consensus.cc:2229] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:54.080765 12110 raft_consensus.cc:2258] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:54.081086 12110 ts_tablet_manager.cc:1905] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:54.082863 12110 ts_tablet_manager.cc:1918] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:54.082929 12110 log.cc:1196] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/a636b870ac0145c69895a371b77bda44
I20260924 07:18:54.083137 12110 ts_tablet_manager.cc:1939] T a636b870ac0145c69895a371b77bda44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Deleting consensus metadata
I20260924 07:18:54.083578 11561 catalog_manager.cc:4841] TS e5e08ed215ad4a76a16a3816f9a7ddfa (127.3.246.66:41327): tablet a636b870ac0145c69895a371b77bda44 (table test-table [id=46573a73c541490ea85f1cbcb1c554f7]) successfully deleted
Sep 24 07:18:54 dist-test-slave-rc1w krb5kdc[11530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:18:54 dist-test-slave-rc1w krb5kdc[11530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:54 dist-test-slave-rc1w krb5kdc[11530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:18:54.111964 11625 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:54.104981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.3.246.126:37141)
0924 07:18:54.105033 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.105036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.105172 (+ 136us) server_negotiation.cc:407] Connection header received
0924 07:18:54.105233 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.105234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.105245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.105323 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:54.106080 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.106853 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.107990 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.108279 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.108370 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.111893 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60054: BlockingRecv error: recv got EOF from 127.0.0.1:60054 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 07:18:54 dist-test-slave-rc1w krb5kdc[11530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM
I20260924 07:18:54.123214 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.113807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.3.246.126:37141)
0924 07:18:54.113845 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.113847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.114122 (+ 275us) server_negotiation.cc:407] Connection header received
0924 07:18:54.114220 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.114221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.114230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.114259 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:54.114884 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.115727 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.116918 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.117206 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.117284 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.120163 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:54.120165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:54.120165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:54.120172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:54.121959 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:54.122380 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:54.122382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:54.122382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:54.122449 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:54.122679 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:54.122680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:54.122681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:54.122825 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:54.122926 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:54.123037 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 07:18:54.123116 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:18:54.125453 11574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:54.125746 11574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:54.127110 11712 tablet_service.cc:1467] Processing CreateTablet for tablet 50fdaa4165af4bde95385def634cbd61 (DEFAULT_TABLE table=test-table [id=109d8eb185134b5a9936afea040d5a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:54.127247 11712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fdaa4165af4bde95385def634cbd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:54.127236 11852 tablet_service.cc:1467] Processing CreateTablet for tablet 50fdaa4165af4bde95385def634cbd61 (DEFAULT_TABLE table=test-table [id=109d8eb185134b5a9936afea040d5a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:54.127383 11852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fdaa4165af4bde95385def634cbd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:54.128608 12088 tablet_bootstrap.cc:492] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec: Bootstrap starting.
I20260924 07:18:54.128803 11991 tablet_service.cc:1467] Processing CreateTablet for tablet 50fdaa4165af4bde95385def634cbd61 (DEFAULT_TABLE table=test-table [id=109d8eb185134b5a9936afea040d5a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:54.128924 11991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fdaa4165af4bde95385def634cbd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:54.128929 12088 tablet_bootstrap.cc:654] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:54.129492 12088 tablet_bootstrap.cc:492] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec: No bootstrap required, opened a new log
I20260924 07:18:54.129532 12088 ts_tablet_manager.cc:1397] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:54.129705 12088 raft_consensus.cc:348] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.129773 12088 raft_consensus.cc:374] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:54.129794 12088 raft_consensus.cc:729] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3fd817383dd477f9ecb3d7a1b30f9ec, State: Initialized, Role: FOLLOWER
I20260924 07:18:54.129834 12088 consensus_queue.cc:260] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.129930 12088 ts_tablet_manager.cc:1428] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:54.130388 12087 tablet_bootstrap.cc:492] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680: Bootstrap starting.
I20260924 07:18:54.130393 12086 tablet_bootstrap.cc:492] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa: Bootstrap starting.
I20260924 07:18:54.130797 12087 tablet_bootstrap.cc:654] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:54.130839 12086 tablet_bootstrap.cc:654] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:54.131528 12086 tablet_bootstrap.cc:492] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa: No bootstrap required, opened a new log
I20260924 07:18:54.131569 12086 ts_tablet_manager.cc:1397] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:54.131713 12086 raft_consensus.cc:348] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.131786 12086 raft_consensus.cc:374] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:54.131807 12086 raft_consensus.cc:729] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e08ed215ad4a76a16a3816f9a7ddfa, State: Initialized, Role: FOLLOWER
I20260924 07:18:54.131847 12086 consensus_queue.cc:260] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.131924 12086 ts_tablet_manager.cc:1428] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:54.133066 12087 tablet_bootstrap.cc:492] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680: No bootstrap required, opened a new log
I20260924 07:18:54.133123 12087 ts_tablet_manager.cc:1397] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 07:18:54.133262 12087 raft_consensus.cc:348] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.133358 12087 raft_consensus.cc:374] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:54.133380 12087 raft_consensus.cc:729] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff0c1f3b43742a99bec47debb039680, State: Initialized, Role: FOLLOWER
I20260924 07:18:54.133427 12087 consensus_queue.cc:260] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.133509 12087 ts_tablet_manager.cc:1428] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:54.471035 12138 raft_consensus.cc:482] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:54.471134 12138 raft_consensus.cc:504] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.471338 12138 leader_election.cc:288] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3fd817383dd477f9ecb3d7a1b30f9ec (127.3.246.65:39661), cff0c1f3b43742a99bec47debb039680 (127.3.246.67:40629)
I20260924 07:18:54.471657 12010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fdaa4165af4bde95385def634cbd61" candidate_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff0c1f3b43742a99bec47debb039680" is_pre_election: true
I20260924 07:18:54.471657 11732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fdaa4165af4bde95385def634cbd61" candidate_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" is_pre_election: true
I20260924 07:18:54.471768 12010 raft_consensus.cc:2454] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e08ed215ad4a76a16a3816f9a7ddfa in term 0.
I20260924 07:18:54.471768 11732 raft_consensus.cc:2454] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e08ed215ad4a76a16a3816f9a7ddfa in term 0.
I20260924 07:18:54.471966 11806 leader_election.cc:302] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3fd817383dd477f9ecb3d7a1b30f9ec, e5e08ed215ad4a76a16a3816f9a7ddfa; no voters:
I20260924 07:18:54.472100 12138 raft_consensus.cc:2781] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:54.472143 12138 raft_consensus.cc:482] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:54.472168 12138 raft_consensus.cc:3037] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:54.472972 12138 raft_consensus.cc:504] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.473119 12138 leader_election.cc:288] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [CANDIDATE]: Term 1 election: Requested vote from peers a3fd817383dd477f9ecb3d7a1b30f9ec (127.3.246.65:39661), cff0c1f3b43742a99bec47debb039680 (127.3.246.67:40629)
I20260924 07:18:54.473259 11732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fdaa4165af4bde95385def634cbd61" candidate_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec"
I20260924 07:18:54.473317 12010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fdaa4165af4bde95385def634cbd61" candidate_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff0c1f3b43742a99bec47debb039680"
I20260924 07:18:54.473371 11732 raft_consensus.cc:3037] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:54.473388 12010 raft_consensus.cc:3037] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:54.474074 11732 raft_consensus.cc:2454] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e08ed215ad4a76a16a3816f9a7ddfa in term 1.
I20260924 07:18:54.474134 12010 raft_consensus.cc:2454] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e08ed215ad4a76a16a3816f9a7ddfa in term 1.
I20260924 07:18:54.474303 11806 leader_election.cc:302] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3fd817383dd477f9ecb3d7a1b30f9ec, e5e08ed215ad4a76a16a3816f9a7ddfa; no voters:
I20260924 07:18:54.474541 12138 raft_consensus.cc:2781] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:54.474629 12138 raft_consensus.cc:686] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 LEADER]: Becoming Leader. State: Replica: e5e08ed215ad4a76a16a3816f9a7ddfa, State: Running, Role: LEADER
I20260924 07:18:54.474686 12138 consensus_queue.cc:237] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.475330 11574 catalog_manager.cc:5495] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e08ed215ad4a76a16a3816f9a7ddfa (127.3.246.66). New cstate: current_term: 1 leader_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } health_report { overall_health: UNKNOWN } } }
I20260924 07:18:54.490182 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.487961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.3.246.66:41327)
0924 07:18:54.488029 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.488031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.488039 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:18:54.488076 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.488076 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.488085 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.488130 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:54.488264 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.488737 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.489474 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.489656 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.489794 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.489942 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:54.489974 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:54.490016 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:54.490068 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 07:18:54.490085 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:18:54.490978 12010 raft_consensus.cc:1261] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Refusing update from remote peer e5e08ed215ad4a76a16a3816f9a7ddfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:54.491045 11732 raft_consensus.cc:1261] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Refusing update from remote peer e5e08ed215ad4a76a16a3816f9a7ddfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:54.491223 12092 consensus_queue.cc:1035] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:54.491313 12138 consensus_queue.cc:1035] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:54.493599 11574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:54.493742 11574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:54.494347 11574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f: Sending DeleteTablet for 3 replicas of tablet 50fdaa4165af4bde95385def634cbd61
I20260924 07:18:54.494601 11712 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fdaa4165af4bde95385def634cbd61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:54 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:35542
I20260924 07:18:54.494624 11991 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fdaa4165af4bde95385def634cbd61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:54 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:34630
I20260924 07:18:54.494611 11852 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fdaa4165af4bde95385def634cbd61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:54 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:34530
I20260924 07:18:54.494710 12109 tablet_replica.cc:330] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680: stopping tablet replica
I20260924 07:18:54.494714 12110 tablet_replica.cc:330] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa: stopping tablet replica
I20260924 07:18:54.494722 12111 tablet_replica.cc:330] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec: stopping tablet replica
I20260924 07:18:54.494763 12109 raft_consensus.cc:2229] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:54.494781 12110 raft_consensus.cc:2229] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:54.494783 12111 raft_consensus.cc:2229] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:54.494808 12109 raft_consensus.cc:2258] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:54.494824 12111 raft_consensus.cc:2258] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:54.494837 12110 raft_consensus.cc:2258] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:54.495193 12111 ts_tablet_manager.cc:1905] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:54.496959 12111 ts_tablet_manager.cc:1918] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:54.497011 12111 log.cc:1196] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-0/wal/wals/50fdaa4165af4bde95385def634cbd61
I20260924 07:18:54.497241 12111 ts_tablet_manager.cc:1939] T 50fdaa4165af4bde95385def634cbd61 P a3fd817383dd477f9ecb3d7a1b30f9ec: Deleting consensus metadata
I20260924 07:18:54.497704 11562 catalog_manager.cc:4841] TS a3fd817383dd477f9ecb3d7a1b30f9ec (127.3.246.65:39661): tablet 50fdaa4165af4bde95385def634cbd61 (table test-table [id=109d8eb185134b5a9936afea040d5a34]) successfully deleted
I20260924 07:18:54.499240 12109 ts_tablet_manager.cc:1905] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:54.499423 12110 ts_tablet_manager.cc:1905] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:54.499603 11574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:54.499748 11574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:54.500735 12109 ts_tablet_manager.cc:1918] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:54.500802 12109 log.cc:1196] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/50fdaa4165af4bde95385def634cbd61
I20260924 07:18:54.501036 12109 ts_tablet_manager.cc:1939] T 50fdaa4165af4bde95385def634cbd61 P cff0c1f3b43742a99bec47debb039680: Deleting consensus metadata
I20260924 07:18:54.501437 11561 catalog_manager.cc:4841] TS cff0c1f3b43742a99bec47debb039680 (127.3.246.67:40629): tablet 50fdaa4165af4bde95385def634cbd61 (table test-table [id=109d8eb185134b5a9936afea040d5a34]) successfully deleted
I20260924 07:18:54.501456 12110 ts_tablet_manager.cc:1918] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:54.501509 12110 log.cc:1196] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/50fdaa4165af4bde95385def634cbd61
I20260924 07:18:54.501758 12110 ts_tablet_manager.cc:1939] T 50fdaa4165af4bde95385def634cbd61 P e5e08ed215ad4a76a16a3816f9a7ddfa: Deleting consensus metadata
I20260924 07:18:54.502326 11712 tablet_service.cc:1467] Processing CreateTablet for tablet ddd575d890f44bbd99b0f556f9029f44 (DEFAULT_TABLE table=test-table [id=c0e523ed90dc4ae29c360878f4c6a7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:54.502336 11991 tablet_service.cc:1467] Processing CreateTablet for tablet ddd575d890f44bbd99b0f556f9029f44 (DEFAULT_TABLE table=test-table [id=c0e523ed90dc4ae29c360878f4c6a7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:54.502367 11852 tablet_service.cc:1467] Processing CreateTablet for tablet ddd575d890f44bbd99b0f556f9029f44 (DEFAULT_TABLE table=test-table [id=c0e523ed90dc4ae29c360878f4c6a7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:54.502439 11991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd575d890f44bbd99b0f556f9029f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:54.502449 11852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd575d890f44bbd99b0f556f9029f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:54.502463 11712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd575d890f44bbd99b0f556f9029f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:54.503345 12088 tablet_bootstrap.cc:492] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec: Bootstrap starting.
I20260924 07:18:54.503597 12087 tablet_bootstrap.cc:492] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680: Bootstrap starting.
I20260924 07:18:54.503609 11561 catalog_manager.cc:4841] TS e5e08ed215ad4a76a16a3816f9a7ddfa (127.3.246.66:41327): tablet 50fdaa4165af4bde95385def634cbd61 (table test-table [id=109d8eb185134b5a9936afea040d5a34]) successfully deleted
I20260924 07:18:54.503609 12086 tablet_bootstrap.cc:492] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Bootstrap starting.
I20260924 07:18:54.503665 12088 tablet_bootstrap.cc:654] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:54.503931 12086 tablet_bootstrap.cc:654] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:54.503959 12087 tablet_bootstrap.cc:654] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:54.504197 12088 tablet_bootstrap.cc:492] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec: No bootstrap required, opened a new log
I20260924 07:18:54.504283 12088 ts_tablet_manager.cc:1397] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:54.504410 12088 raft_consensus.cc:348] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.504465 12086 tablet_bootstrap.cc:492] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa: No bootstrap required, opened a new log
I20260924 07:18:54.504498 12088 raft_consensus.cc:374] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:54.504511 12086 ts_tablet_manager.cc:1397] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:54.504523 12088 raft_consensus.cc:729] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3fd817383dd477f9ecb3d7a1b30f9ec, State: Initialized, Role: FOLLOWER
I20260924 07:18:54.504576 12088 consensus_queue.cc:260] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.504609 12086 raft_consensus.cc:348] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.504664 12086 raft_consensus.cc:374] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:54.504667 12088 ts_tablet_manager.cc:1428] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:54.504686 12086 raft_consensus.cc:729] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e08ed215ad4a76a16a3816f9a7ddfa, State: Initialized, Role: FOLLOWER
I20260924 07:18:54.504730 12086 consensus_queue.cc:260] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.504835 12086 ts_tablet_manager.cc:1428] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:54.505275 12087 tablet_bootstrap.cc:492] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680: No bootstrap required, opened a new log
I20260924 07:18:54.505322 12087 ts_tablet_manager.cc:1397] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260924 07:18:54.505461 12087 raft_consensus.cc:348] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.505541 12087 raft_consensus.cc:374] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:54.505573 12087 raft_consensus.cc:729] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff0c1f3b43742a99bec47debb039680, State: Initialized, Role: FOLLOWER
I20260924 07:18:54.505626 12087 consensus_queue.cc:260] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.505769 12087 ts_tablet_manager.cc:1428] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:54.552163 12093 raft_consensus.cc:482] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:54.552279 12093 raft_consensus.cc:504] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.552664 12093 leader_election.cc:288] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3fd817383dd477f9ecb3d7a1b30f9ec (127.3.246.65:39661), e5e08ed215ad4a76a16a3816f9a7ddfa (127.3.246.66:41327)
I20260924 07:18:54.555810 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.553088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:44705 (local address 127.3.246.66:41327)
0924 07:18:54.553131 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.553133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.553142 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:54.553174 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.553175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.553182 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.553260 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:54.553451 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.553983 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.555029 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.555437 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.555497 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.555504 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:54.555690 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 07:18:54.555728 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:18:54.555816 12165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.552824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:41327 (local address 127.3.246.67:44705)
0924 07:18:54.552924 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:54.552938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:54.552993 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:54.553269 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:54.553270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:54.553272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:54.553367 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.553369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.554085 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.554086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.554891 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.554893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.555540 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.555541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.555616 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.555621 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:18:54.555676 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 07:18:54.555714 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":4,"thread_start_us":57,"threads_started":1}
I20260924 07:18:54.555987 12166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.552866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:39661 (local address 127.3.246.67:48809)
0924 07:18:54.552972 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:54.552985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:54.553039 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:54.553265 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:54.553267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:54.553268 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:54.553359 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.553361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.554105 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.554106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.555026 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.555028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.555630 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.555630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.555719 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.555723 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:18:54.555781 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 07:18:54.555828 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":119,"thread_start_us":53,"threads_started":1}
I20260924 07:18:54.556041 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd575d890f44bbd99b0f556f9029f44" candidate_uuid: "cff0c1f3b43742a99bec47debb039680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" is_pre_election: true
I20260924 07:18:54.556137 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.553014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:48809 (local address 127.3.246.65:39661)
0924 07:18:54.553137 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.553139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.553145 (+ 6us) server_negotiation.cc:407] Connection header received
0924 07:18:54.553172 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.553173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.553182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.553263 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:54.553441 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.554004 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.555116 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.555545 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.555619 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.555628 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:54.555808 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 07:18:54.556014 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260924 07:18:54.556169 11872 raft_consensus.cc:2454] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff0c1f3b43742a99bec47debb039680 in term 0.
I20260924 07:18:54.556427 11732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd575d890f44bbd99b0f556f9029f44" candidate_uuid: "cff0c1f3b43742a99bec47debb039680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" is_pre_election: true
I20260924 07:18:54.556434 11945 leader_election.cc:302] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cff0c1f3b43742a99bec47debb039680, e5e08ed215ad4a76a16a3816f9a7ddfa; no voters:
I20260924 07:18:54.556551 11732 raft_consensus.cc:2454] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff0c1f3b43742a99bec47debb039680 in term 0.
I20260924 07:18:54.556571 12093 raft_consensus.cc:2781] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:54.556608 12093 raft_consensus.cc:482] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:54.556638 12093 raft_consensus.cc:3037] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:54.557320 12093 raft_consensus.cc:504] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.557472 12093 leader_election.cc:288] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [CANDIDATE]: Term 1 election: Requested vote from peers a3fd817383dd477f9ecb3d7a1b30f9ec (127.3.246.65:39661), e5e08ed215ad4a76a16a3816f9a7ddfa (127.3.246.66:41327)
I20260924 07:18:54.557621 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd575d890f44bbd99b0f556f9029f44" candidate_uuid: "cff0c1f3b43742a99bec47debb039680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa"
I20260924 07:18:54.557636 11732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd575d890f44bbd99b0f556f9029f44" candidate_uuid: "cff0c1f3b43742a99bec47debb039680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec"
I20260924 07:18:54.557749 11872 raft_consensus.cc:3037] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:54.557750 11732 raft_consensus.cc:3037] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:54.558369 11872 raft_consensus.cc:2454] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff0c1f3b43742a99bec47debb039680 in term 1.
I20260924 07:18:54.558455 11732 raft_consensus.cc:2454] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff0c1f3b43742a99bec47debb039680 in term 1.
I20260924 07:18:54.558555 11945 leader_election.cc:302] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cff0c1f3b43742a99bec47debb039680, e5e08ed215ad4a76a16a3816f9a7ddfa; no voters:
I20260924 07:18:54.558637 12093 raft_consensus.cc:2781] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:54.558746 12093 raft_consensus.cc:686] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [term 1 LEADER]: Becoming Leader. State: Replica: cff0c1f3b43742a99bec47debb039680, State: Running, Role: LEADER
I20260924 07:18:54.558821 12093 consensus_queue.cc:237] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } }
I20260924 07:18:54.559551 11574 catalog_manager.cc:5495] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 reported cstate change: term changed from 0 to 1, leader changed from <none> to cff0c1f3b43742a99bec47debb039680 (127.3.246.67). New cstate: current_term: 1 leader_uuid: "cff0c1f3b43742a99bec47debb039680" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff0c1f3b43742a99bec47debb039680" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40629 } health_report { overall_health: HEALTHY } } }
I20260924 07:18:54.569347 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.566806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.3.246.67:40629)
0924 07:18:54.566945 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.566947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.566952 (+ 5us) server_negotiation.cc:407] Connection header received
0924 07:18:54.566982 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.566983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.566998 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.567062 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:54.567281 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.567787 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.568517 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.568732 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.568796 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.569029 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:54.569138 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:54.569211 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:54.569256 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 07:18:54.569274 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260924 07:18:54.574712 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.572150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:40051 (local address 127.3.246.65:39661)
0924 07:18:54.572231 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.572234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.572300 (+ 66us) server_negotiation.cc:407] Connection header received
0924 07:18:54.572364 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.572365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.572377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.572423 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:54.572560 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.573054 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.573971 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.574180 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.574258 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.574445 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:54.574481 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:54.574546 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:54.574605 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 07:18:54.574630 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:18:54.574729 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.572041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:39661 (local address 127.3.246.67:40051)
0924 07:18:54.572221 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:54.572234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:54.572300 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:54.572423 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:54.572424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:54.572425 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:18:54.572504 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.572506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.573145 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.573146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.573873 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.573876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.574280 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.574281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.574350 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.574362 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:18:54.574560 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:18:54.574562 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:18:54.574604 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 07:18:54.574651 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":90,"threads_started":1}
I20260924 07:18:54.577039 11872 raft_consensus.cc:1261] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Refusing update from remote peer cff0c1f3b43742a99bec47debb039680: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:54.577086 11732 raft_consensus.cc:1261] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Refusing update from remote peer cff0c1f3b43742a99bec47debb039680: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:54.577318 12168 consensus_queue.cc:1035] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3fd817383dd477f9ecb3d7a1b30f9ec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 39661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:54.577455 12093 consensus_queue.cc:1035] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e08ed215ad4a76a16a3816f9a7ddfa" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:54.584545 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.582198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:51259 (local address 127.3.246.67:40629)
0924 07:18:54.582236 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.582238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.582336 (+ 98us) server_negotiation.cc:407] Connection header received
0924 07:18:54.582424 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.582425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.582437 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.582489 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:54.582594 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.583100 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.583816 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.584087 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.584154 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.584299 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:54.584331 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:54.584396 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:54.584475 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 07:18:54.584498 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:18:54.584573 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.582165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:40629 (local address 127.3.246.65:51259)
0924 07:18:54.582282 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:54.582292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:54.582371 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:54.582503 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:54.582504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:54.582505 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:18:54.582554 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.582555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.583162 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.583163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.583735 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.583737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.584155 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.584156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.584218 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.584223 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:18:54.584406 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:18:54.584408 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:18:54.584464 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 07:18:54.584520 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":47,"threads_started":1}
I20260924 07:18:54.590907 11574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:54.591013 11574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:54.591638 11574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cbef49b7b1d4b6391b3cdae6bb0118f: Sending DeleteTablet for 3 replicas of tablet ddd575d890f44bbd99b0f556f9029f44
I20260924 07:18:54.591917 11852 tablet_service.cc:1514] Processing DeleteTablet for tablet ddd575d890f44bbd99b0f556f9029f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:54 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:34530
I20260924 07:18:54.591921 11991 tablet_service.cc:1514] Processing DeleteTablet for tablet ddd575d890f44bbd99b0f556f9029f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:54 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:34630
I20260924 07:18:54.592015 12110 tablet_replica.cc:330] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa: stopping tablet replica
I20260924 07:18:54.592015 11712 tablet_service.cc:1514] Processing DeleteTablet for tablet ddd575d890f44bbd99b0f556f9029f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:54 UTC) from {username='oryx', principal='oryx/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:35542
I20260924 07:18:54.592068 12110 raft_consensus.cc:2229] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:54.592084 12111 tablet_replica.cc:330] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec: stopping tablet replica
I20260924 07:18:54.592104 12110 raft_consensus.cc:2258] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:54.592133 12111 raft_consensus.cc:2229] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:54.592167 12111 raft_consensus.cc:2258] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:54.592170 12109 tablet_replica.cc:330] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680: stopping tablet replica
I20260924 07:18:54.592217 12109 raft_consensus.cc:2229] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:54.592303 12109 raft_consensus.cc:2258] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:54.592597 12109 ts_tablet_manager.cc:1905] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:54.592588 12111 ts_tablet_manager.cc:1905] T ddd575d890f44bbd99b0f556f9029f44 P a3fd817383dd477f9ecb3d7a1b30f9ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:54.592613 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 11647
I20260924 07:18:54.593701 12110 ts_tablet_manager.cc:1905] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:54.594543 12109 ts_tablet_manager.cc:1918] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:54.594622 12109 log.cc:1196] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/ddd575d890f44bbd99b0f556f9029f44
I20260924 07:18:54.594842 12109 ts_tablet_manager.cc:1939] T ddd575d890f44bbd99b0f556f9029f44 P cff0c1f3b43742a99bec47debb039680: Deleting consensus metadata
I20260924 07:18:54.595293 12110 ts_tablet_manager.cc:1918] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:18:54.595372 12110 log.cc:1196] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/ddd575d890f44bbd99b0f556f9029f44
I20260924 07:18:54.595476 11561 catalog_manager.cc:4841] TS cff0c1f3b43742a99bec47debb039680 (127.3.246.67:40629): tablet ddd575d890f44bbd99b0f556f9029f44 (table test-table [id=c0e523ed90dc4ae29c360878f4c6a7e2]) successfully deleted
I20260924 07:18:54.595590 12110 ts_tablet_manager.cc:1939] T ddd575d890f44bbd99b0f556f9029f44 P e5e08ed215ad4a76a16a3816f9a7ddfa: Deleting consensus metadata
I20260924 07:18:54.596086 11561 catalog_manager.cc:4841] TS e5e08ed215ad4a76a16a3816f9a7ddfa (127.3.246.66:41327): tablet ddd575d890f44bbd99b0f556f9029f44 (table test-table [id=c0e523ed90dc4ae29c360878f4c6a7e2]) successfully deleted
W20260924 07:18:54.599970 11562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:39661 (error 108)
W20260924 07:18:54.600512 12183 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:54.600325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:39661 (local address 127.0.0.1:35562)
0924 07:18:54.600425 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:54.600467 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:39661: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":61,"threads_started":1}
W20260924 07:18:54.600606 11562 catalog_manager.cc:4584] TS a3fd817383dd477f9ecb3d7a1b30f9ec (127.3.246.65:39661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ddd575d890f44bbd99b0f556f9029f44: Network error: Client connection negotiation failed: client connection to 127.3.246.65:39661: connect: Connection reset by peer (error 104)
I20260924 07:18:54.600890 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 11787
I20260924 07:18:54.606958 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 11927
I20260924 07:18:54.612473 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 11543
2026-09-24T07:18:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3002 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:54 dist-test-slave-rc1w krb5kdc[12186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:54 dist-test-slave-rc1w krb5kdc[12186](info): set up 2 sockets
Sep 24 07:18:54 dist-test-slave-rc1w krb5kdc[12186](info): commencing operation
krb5kdc: starting...
W20260924 07:18:55.650303 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:55 dist-test-slave-rc1w krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:18:55.768841 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:36789
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:36789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:55.848503 12199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:55.848697 12199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:55.848726 12199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:55.849932 12199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:55.849972 12199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:55.849985 12199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:55.849998 12199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:55.850064 12199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:55.851550 12199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:36789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:36789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:55.851822 12199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:55.852065 12199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:55.856842 12207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:55.856812 12204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:55.857090 12199 server_base.cc:1034] running on GCE node
W20260924 07:18:55.856810 12205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:55.857419 12199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:55.857764 12199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:55.858946 12199 hybrid_clock.cc:648] HybridClock initialized: now 1790234335858936 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T07:18:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1252 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:55 dist-test-slave-rc1w krb5kdc[12212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:55 dist-test-slave-rc1w krb5kdc[12212](info): set up 2 sockets
Sep 24 07:18:55 dist-test-slave-rc1w krb5kdc[12212](info): commencing operation
krb5kdc: starting...
W20260924 07:18:56.910248 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:18:56 dist-test-slave-rc1w krb5kdc[12212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:18:57.026677 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:35777
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:35777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:57.105994 12225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:57.106217 12225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:57.106253 12225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:57.107393 12225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:57.107460 12225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:57.107486 12225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:57.107506 12225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:57.107535 12225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:57.108964 12225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:35777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:35777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:57.109230 12225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:57.109460 12225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:57.114073 12230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:57.114080 12231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:57.114080 12233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:57.114316 12225 server_base.cc:1034] running on GCE node
I20260924 07:18:57.114563 12225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:57.114852 12225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:57.116012 12225 hybrid_clock.cc:648] HybridClock initialized: now 1790234337115999 us; error 29 us; skew 500 ppm
Sep 24 07:18:57 dist-test-slave-rc1w krb5kdc[12212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:57.128093 12225 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:18:57.128571 12225 webserver.cc:460] Webserver started at http://127.3.246.126:46585/ using document root <none> and password file <none>
I20260924 07:18:57.128758 12225 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:57.128800 12225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:57.128888 12225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:57.129978 12225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "c005cbac9a6f49e3ab22d5d6472d03e6"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "f4dd60f21b8211e1190653b4ae788f74"
server_key_iv: "d511d4db0ee0fa6b25c84cd68e052ec5"
server_key_version: "encryptionkey@0"
I20260924 07:18:57.130285 12225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "c005cbac9a6f49e3ab22d5d6472d03e6"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "f4dd60f21b8211e1190653b4ae788f74"
server_key_iv: "d511d4db0ee0fa6b25c84cd68e052ec5"
server_key_version: "encryptionkey@0"
I20260924 07:18:57.131520 12225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:57.132211 12240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:57.132396 12225 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:57.132467 12225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "c005cbac9a6f49e3ab22d5d6472d03e6"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "f4dd60f21b8211e1190653b4ae788f74"
server_key_iv: "d511d4db0ee0fa6b25c84cd68e052ec5"
server_key_version: "encryptionkey@0"
I20260924 07:18:57.132519 12225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:57.151183 12225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:57.153019 12225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:57.153127 12225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:57.157029 12225 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:35777
I20260924 07:18:57.157090 12292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:35777 every 8 connection(s)
I20260924 07:18:57.157440 12225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:57.158298 12293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:57.160564 12293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6: Bootstrap starting.
I20260924 07:18:57.161142 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 12225
I20260924 07:18:57.161257 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:18:57.161260 12293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:57.161495 4057 external_mini_cluster.cc:1428] Setting key def74ad831a83bcb332c799e8452a55e
I20260924 07:18:57.161588 12293 log.cc:824] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:57.162436 12293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6: No bootstrap required, opened a new log
I20260924 07:18:57.164904 12293 raft_consensus.cc:348] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35777 } }
I20260924 07:18:57.165066 12293 raft_consensus.cc:374] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:57.165100 12293 raft_consensus.cc:729] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c005cbac9a6f49e3ab22d5d6472d03e6, State: Initialized, Role: FOLLOWER
I20260924 07:18:57.165330 12293 consensus_queue.cc:260] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35777 } }
I20260924 07:18:57.165442 12293 raft_consensus.cc:388] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:57.165488 12293 raft_consensus.cc:482] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:57.165536 12293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:57.166188 12293 raft_consensus.cc:504] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35777 } }
I20260924 07:18:57.166298 12293 leader_election.cc:302] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c005cbac9a6f49e3ab22d5d6472d03e6; no voters:
I20260924 07:18:57.166455 12293 leader_election.cc:288] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:57.166519 12298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:57.166684 12293 sys_catalog.cc:564] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:57.166692 12298 raft_consensus.cc:686] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 1 LEADER]: Becoming Leader. State: Replica: c005cbac9a6f49e3ab22d5d6472d03e6, State: Running, Role: LEADER
I20260924 07:18:57.166908 12298 consensus_queue.cc:237] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35777 } }
I20260924 07:18:57.167258 12299 sys_catalog.cc:455] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35777 } } }
I20260924 07:18:57.167346 12299 sys_catalog.cc:458] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:57.167477 12300 sys_catalog.cc:455] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c005cbac9a6f49e3ab22d5d6472d03e6. Latest consensus state: current_term: 1 leader_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35777 } } }
I20260924 07:18:57.167565 12300 sys_catalog.cc:458] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:57.168309 12307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:57.168941 12307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 07:18:57 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:57.170600 12307 catalog_manager.cc:1347] Generated new cluster ID: 6b5e9df25c484ac0a8aecb6b79c3e63c
I20260924 07:18:57.170651 12307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:57.173312 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:57.162047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.3.246.126:35777)
0924 07:18:57.162194 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:57.162203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:57.162239 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:18:57.162834 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:57.162844 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:57.162907 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:57.163060 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:57.163639 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.165089 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.166015 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.166645 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.167089 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:57.169480 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:57.169484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:57.169484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:57.169496 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:57.171690 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:57.172121 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:57.172124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:57.172125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:57.172216 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:57.172451 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:57.172452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:57.172453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:57.172684 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:57.172851 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:57.172960 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 07:18:57.173092 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260924 07:18:57.194615 12307 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:57.195228 12307 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:57.200388 12307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6: Generated new TSK 0
I20260924 07:18:57.200587 12307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:18:57.232549 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:35777
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:57.332788 12319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:57.332974 12319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:57.332998 12319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:57.334095 12319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:57.334128 12319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:57.334163 12319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:57.335522 12319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:35777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:57.335691 12319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:57.335914 12319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:57.340731 12327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:57.340888 12319 server_base.cc:1034] running on GCE node
W20260924 07:18:57.340754 12324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:57.340754 12325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:57.341187 12319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:57.341384 12319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:57.342521 12319 hybrid_clock.cc:648] HybridClock initialized: now 1790234337342510 us; error 28 us; skew 500 ppm
Sep 24 07:18:57 dist-test-slave-rc1w krb5kdc[12212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:57.344928 12319 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:18:57.345331 12319 webserver.cc:460] Webserver started at http://127.3.246.65:41141/ using document root <none> and password file <none>
I20260924 07:18:57.345564 12319 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:57.345633 12319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:57.345809 12319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:57.346830 12319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "5fd3e946a97d4daab266598c2c350850"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "330471946710ed69510741e22399dafb"
server_key_iv: "5f134596163414e6142e32c8d9680039"
server_key_version: "encryptionkey@0"
I20260924 07:18:57.347187 12319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "5fd3e946a97d4daab266598c2c350850"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "330471946710ed69510741e22399dafb"
server_key_iv: "5f134596163414e6142e32c8d9680039"
server_key_version: "encryptionkey@0"
I20260924 07:18:57.348493 12319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:18:57.349198 12334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:57.349481 12319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:57.349571 12319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "5fd3e946a97d4daab266598c2c350850"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "330471946710ed69510741e22399dafb"
server_key_iv: "5f134596163414e6142e32c8d9680039"
server_key_version: "encryptionkey@0"
I20260924 07:18:57.349651 12319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:57.372967 12319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:57.375418 12319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:57.375581 12319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:57.375825 12319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:57.376130 12319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:57.376190 12319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:57.376231 12319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:57.376291 12319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:57.382426 12319 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:37103
I20260924 07:18:57.382522 12447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:37103 every 8 connection(s)
I20260924 07:18:57.382831 12319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:57.387938 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 12319
I20260924 07:18:57.388023 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:57.388247 4057 external_mini_cluster.cc:1428] Setting key 192e5bbe4d3ac7437b2d6bc809b3f0d1
Sep 24 07:18:57 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:57.392810 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:57.383527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:44429 (local address 127.3.246.126:35777)
0924 07:18:57.383593 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:57.383594 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:57.383800 (+ 206us) server_negotiation.cc:407] Connection header received
0924 07:18:57.384359 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:57.384360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:57.384373 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:57.384419 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:57.385357 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.385953 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.386957 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.387177 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.387263 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:57.389751 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:57.389752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:57.389753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:57.389760 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:57.391419 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:57.391793 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:57.391794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:57.391795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:57.391859 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:57.392103 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:57.392103 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:57.392104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:57.392248 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:57.392326 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:57.392558 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 07:18:57.392661 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:18:57.392884 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:57.383498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:44429)
0924 07:18:57.383685 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:57.383720 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:57.384231 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:57.384504 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:57.384513 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:57.385068 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:57.385249 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:57.385253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.386045 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.386046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:57.386863 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:57.386866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.387274 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.387275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:57.387395 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:57.387798 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:57.387806 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:57.389599 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:57.391498 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:57.391505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:57.391506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:57.391717 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:57.391926 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:57.391926 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:57.391927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:57.392038 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:57.392312 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:57.392317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:57.392427 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 07:18:57.392507 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:18:57.392682 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260924 07:18:57.393352 12448 heartbeater.cc:347] Connected to a master server at 127.3.246.126:35777
I20260924 07:18:57.393453 12448 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:57.393651 12448 heartbeater.cc:511] Master 127.3.246.126:35777 requested a full tablet report, sending...
I20260924 07:18:57.394289 12257 ts_manager.cc:194] Registered new tserver with Master: 5fd3e946a97d4daab266598c2c350850 (127.3.246.65:37103)
I20260924 07:18:57.395228 12257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:44429
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:18:57.413307 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:35777
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:57.491901 12453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:57.492096 12453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:57.492120 12453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:57.493198 12453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:57.493232 12453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:57.493260 12453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:57.494596 12453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:35777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:57.494793 12453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:57.495028 12453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:18:57.499985 12453 server_base.cc:1034] running on GCE node
W20260924 07:18:57.499915 12458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:57.499890 12459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:57.499895 12461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:57.500367 12453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:57.500634 12453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:57.501744 12453 hybrid_clock.cc:648] HybridClock initialized: now 1790234337501718 us; error 40 us; skew 500 ppm
Sep 24 07:18:57 dist-test-slave-rc1w krb5kdc[12212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:57.504076 12453 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:18:57.504534 12453 webserver.cc:460] Webserver started at http://127.3.246.66:36659/ using document root <none> and password file <none>
I20260924 07:18:57.504755 12453 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:57.504832 12453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:57.504963 12453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:57.506048 12453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "c3840bf65437419eb7eb736601dc219c"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "4e96790a9d58be7ab5dbeea286a71f4e"
server_key_iv: "21fae5c4e8b143cd435fff49f3177aa2"
server_key_version: "encryptionkey@0"
I20260924 07:18:57.506398 12453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "c3840bf65437419eb7eb736601dc219c"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "4e96790a9d58be7ab5dbeea286a71f4e"
server_key_iv: "21fae5c4e8b143cd435fff49f3177aa2"
server_key_version: "encryptionkey@0"
I20260924 07:18:57.507678 12453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:57.508354 12468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:57.508528 12453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:57.508623 12453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "c3840bf65437419eb7eb736601dc219c"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "4e96790a9d58be7ab5dbeea286a71f4e"
server_key_iv: "21fae5c4e8b143cd435fff49f3177aa2"
server_key_version: "encryptionkey@0"
I20260924 07:18:57.508708 12453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:57.533592 12453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:57.535482 12453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:57.535645 12453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:57.535882 12453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:57.536186 12453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:57.536240 12453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:57.536274 12453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:57.536303 12453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:57.542124 12453 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:34847
I20260924 07:18:57.542172 12581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:34847 every 8 connection(s)
I20260924 07:18:57.542467 12453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:57.547818 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 12453
I20260924 07:18:57.547928 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:57.548142 4057 external_mini_cluster.cc:1428] Setting key 64bc5320b77294509ff1c488ac8d3564
Sep 24 07:18:57 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:57.551545 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:57.543171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:35057 (local address 127.3.246.126:35777)
0924 07:18:57.543234 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:57.543236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:57.543454 (+ 218us) server_negotiation.cc:407] Connection header received
0924 07:18:57.543908 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:57.543909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:57.543918 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:57.543974 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:57.544828 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.545309 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.546281 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.546468 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.546522 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:57.548707 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:57.548709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:57.548709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:57.548717 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:57.550141 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:57.550555 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:57.550557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:57.550557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:57.550620 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:57.550893 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:57.550894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:57.550895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:57.551041 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:57.551102 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:57.551421 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 07:18:57.551482 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:18:57.551847 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:57.543162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:35057)
0924 07:18:57.543319 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:57.543360 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:57.543826 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:57.544024 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:57.544027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:57.544561 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:57.544748 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:57.544753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.545392 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.545392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:57.546208 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:57.546210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.546542 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.546542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:57.546640 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:57.547063 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:57.547071 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:57.548606 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:57.550242 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:57.550249 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:57.550251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:57.550478 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:57.550693 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:57.550694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:57.550694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:57.550800 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:57.551110 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:57.551116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:57.551282 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 07:18:57.551396 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 07:18:57.551569 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260924 07:18:57.552356 12582 heartbeater.cc:347] Connected to a master server at 127.3.246.126:35777
I20260924 07:18:57.552464 12582 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:57.552659 12582 heartbeater.cc:511] Master 127.3.246.126:35777 requested a full tablet report, sending...
I20260924 07:18:57.553081 12257 ts_manager.cc:194] Registered new tserver with Master: c3840bf65437419eb7eb736601dc219c (127.3.246.66:34847)
I20260924 07:18:57.553457 12257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:35057
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:18:57.573158 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:35777
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:57.654757 12587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:57.654970 12587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:57.654994 12587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:57.656211 12587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:57.656255 12587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:57.656284 12587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:57.657860 12587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:35777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:57.658097 12587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:57.658342 12587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:57.663470 12592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:57.663465 12593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:57.663565 12587 server_base.cc:1034] running on GCE node
W20260924 07:18:57.663501 12595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:57.663884 12587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:57.664085 12587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:57.665269 12587 hybrid_clock.cc:648] HybridClock initialized: now 1790234337665246 us; error 32 us; skew 500 ppm
Sep 24 07:18:57 dist-test-slave-rc1w krb5kdc[12212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:57.668128 12587 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:18:57.668610 12587 webserver.cc:460] Webserver started at http://127.3.246.67:42997/ using document root <none> and password file <none>
I20260924 07:18:57.668808 12587 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:57.668855 12587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:57.668946 12587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:57.670042 12587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "0f71a7e1d2894dc6839ff6fae2248137"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "95889eae4069f3a61339d6f47077deed"
server_key_iv: "9d768a5636d886bec2666e187455551d"
server_key_version: "encryptionkey@0"
I20260924 07:18:57.670346 12587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "0f71a7e1d2894dc6839ff6fae2248137"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "95889eae4069f3a61339d6f47077deed"
server_key_iv: "9d768a5636d886bec2666e187455551d"
server_key_version: "encryptionkey@0"
I20260924 07:18:57.671591 12587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:57.672279 12602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:57.672466 12587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:18:57.672572 12587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "0f71a7e1d2894dc6839ff6fae2248137"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "95889eae4069f3a61339d6f47077deed"
server_key_iv: "9d768a5636d886bec2666e187455551d"
server_key_version: "encryptionkey@0"
I20260924 07:18:57.672639 12587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:57.705328 12587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:57.707487 12587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:57.707638 12587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:57.707901 12587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:57.708223 12587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:57.708256 12587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:57.708285 12587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:57.708298 12587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:57.714322 12587 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:33873
I20260924 07:18:57.714365 12715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:33873 every 8 connection(s)
I20260924 07:18:57.714696 12587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:57.718096 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 12587
I20260924 07:18:57.718180 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:57.718365 4057 external_mini_cluster.cc:1428] Setting key bfa2b4846a43d98c3913fcde5a5df4c7
Sep 24 07:18:57 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:18:57.724498 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:57.715453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:59047 (local address 127.3.246.126:35777)
0924 07:18:57.715522 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:57.715524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:57.715726 (+ 202us) server_negotiation.cc:407] Connection header received
0924 07:18:57.716219 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:57.716220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:57.716231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:57.716302 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:57.717265 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.717849 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.719343 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.719531 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.719585 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:57.721793 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:57.721795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:57.721795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:57.721802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:57.723078 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:57.723472 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:57.723472 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:57.723473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:57.723534 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:57.723807 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:57.723808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:57.723809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:57.723969 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:57.724037 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:57.724353 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 07:18:57.724423 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:18:57.724632 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:57.715436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:59047)
0924 07:18:57.715600 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:57.715640 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:57.716122 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:57.716357 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:57.716361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:57.716957 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:57.717152 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:57.717157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.717948 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.717949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:57.719261 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:57.719264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:57.719599 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:57.719600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:57.719702 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:57.720301 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:57.720310 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:57.721656 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:57.723171 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:57.723177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:57.723179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:57.723397 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:57.723599 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:57.723600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:57.723600 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:57.723691 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:57.724077 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:57.724083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:57.724199 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 07:18:57.724302 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 07:18:57.724424 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260924 07:18:57.725147 12716 heartbeater.cc:347] Connected to a master server at 127.3.246.126:35777
I20260924 07:18:57.725241 12716 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:57.725438 12716 heartbeater.cc:511] Master 127.3.246.126:35777 requested a full tablet report, sending...
I20260924 07:18:57.725845 12257 ts_manager.cc:194] Registered new tserver with Master: 0f71a7e1d2894dc6839ff6fae2248137 (127.3.246.67:33873)
I20260924 07:18:57.726220 12257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:59047
I20260924 07:18:57.730008 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab.
Entry for principal oryx/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab.
WARNING: no policy specified for oryx/127.3.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab.
Entry for principal oryx/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab.
WARNING: no policy specified for oryx/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab.
Entry for principal oryx/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab.
I20260924 07:18:57.800372 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 12319
I20260924 07:18:57.804826 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 12453
I20260924 07:18:57.809046 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 12587
I20260924 07:18:57.813301 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 12225
I20260924 07:18:57.818439 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:35777
--webserver_interface=127.3.246.126
--webserver_port=46585
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:35777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:57.899770 12725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:57.900019 12725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:57.900056 12725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:57.901202 12725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:57.901295 12725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:57.901320 12725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:57.901347 12725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:57.901379 12725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:57.902853 12725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:35777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:35777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=46585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:57.903117 12725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:57.903342 12725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:57.908169 12731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:57.908174 12730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:57.908202 12733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:57.908282 12725 server_base.cc:1034] running on GCE node
I20260924 07:18:57.908596 12725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:57.908897 12725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:57.910075 12725 hybrid_clock.cc:648] HybridClock initialized: now 1790234337910063 us; error 27 us; skew 500 ppm
Sep 24 07:18:57 dist-test-slave-rc1w krb5kdc[12212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:57.912636 12725 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:18:57.913115 12725 webserver.cc:460] Webserver started at http://127.3.246.126:46585/ using document root <none> and password file <none>
I20260924 07:18:57.913360 12725 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:57.913445 12725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:57.914901 12725 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:57.915671 12740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:57.915884 12725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:57.916000 12725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "c005cbac9a6f49e3ab22d5d6472d03e6"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "f4dd60f21b8211e1190653b4ae788f74"
server_key_iv: "d511d4db0ee0fa6b25c84cd68e052ec5"
server_key_version: "encryptionkey@0"
I20260924 07:18:57.916406 12725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:57.926959 12725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:57.929425 12725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:57.929605 12725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:57.933938 12725 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:35777
I20260924 07:18:57.933981 12792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:35777 every 8 connection(s)
I20260924 07:18:57.934398 12725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:57.935877 12793 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:18:57.936465 12793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6: Bootstrap starting.
I20260924 07:18:57.943195 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 12725
I20260924 07:18:57.943534 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:37103
--local_ip_for_outbound_sockets=127.3.246.65
--tserver_master_addrs=127.3.246.126:35777
--webserver_port=41141
--webserver_interface=127.3.246.65
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.246.65
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:18:57.945261 12793 log.cc:824] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:57.947712 12793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:18:57.948019 12793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6: Bootstrap complete.
I20260924 07:18:57.951259 12793 raft_consensus.cc:348] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35777 } }
I20260924 07:18:57.951607 12793 raft_consensus.cc:729] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c005cbac9a6f49e3ab22d5d6472d03e6, State: Initialized, Role: FOLLOWER
I20260924 07:18:57.951828 12793 consensus_queue.cc:260] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35777 } }
I20260924 07:18:57.951943 12793 raft_consensus.cc:388] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:57.952029 12793 raft_consensus.cc:482] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:57.952132 12793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:18:57.953533 12793 raft_consensus.cc:504] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35777 } }
I20260924 07:18:57.953619 12793 leader_election.cc:302] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c005cbac9a6f49e3ab22d5d6472d03e6; no voters:
I20260924 07:18:57.953820 12793 leader_election.cc:288] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:18:57.953855 12797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:18:57.953984 12797 raft_consensus.cc:686] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [term 2 LEADER]: Becoming Leader. State: Replica: c005cbac9a6f49e3ab22d5d6472d03e6, State: Running, Role: LEADER
I20260924 07:18:57.954115 12793 sys_catalog.cc:564] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:57.954140 12797 consensus_queue.cc:237] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35777 } }
I20260924 07:18:57.954429 12797 sys_catalog.cc:455] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c005cbac9a6f49e3ab22d5d6472d03e6. Latest consensus state: current_term: 2 leader_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35777 } } }
I20260924 07:18:57.954516 12797 sys_catalog.cc:458] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:57.954434 12798 sys_catalog.cc:455] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c005cbac9a6f49e3ab22d5d6472d03e6" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35777 } } }
I20260924 07:18:57.954703 12803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:57.954708 12798 sys_catalog.cc:458] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:57.955237 12803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:57.955404 12803 catalog_manager.cc:1259] Loaded cluster ID: 6b5e9df25c484ac0a8aecb6b79c3e63c
I20260924 07:18:57.955637 12803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:57.956449 12803 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:57.956645 12803 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c005cbac9a6f49e3ab22d5d6472d03e6: Loaded TSK: 0
I20260924 07:18:57.956869 12803 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:18:58.040614 12795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:58.040814 12795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:58.040838 12795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:58.042039 12795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:58.042078 12795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:58.042104 12795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:18:58.043524 12795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.65.keytab
--principal=oryx/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:37103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=41141
--tserver_master_addrs=127.3.246.126:35777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:58.043787 12795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:58.044011 12795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:58.049947 12819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:58.049958 12820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:58.049998 12822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:58.050863 12795 server_base.cc:1034] running on GCE node
I20260924 07:18:58.051054 12795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:58.051319 12795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:58.052492 12795 hybrid_clock.cc:648] HybridClock initialized: now 1790234338052477 us; error 31 us; skew 500 ppm
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234338, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:58.055362 12795 init.cc:377] Logged in from keytab as oryx/127.3.246.65@KRBTEST.COM (short username oryx)
I20260924 07:18:58.055895 12795 webserver.cc:460] Webserver started at http://127.3.246.65:41141/ using document root <none> and password file <none>
I20260924 07:18:58.056128 12795 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:58.056217 12795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:58.057723 12795 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:18:58.058398 12829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:58.058576 12795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:18:58.058676 12795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "5fd3e946a97d4daab266598c2c350850"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "330471946710ed69510741e22399dafb"
server_key_iv: "5f134596163414e6142e32c8d9680039"
server_key_version: "encryptionkey@0"
I20260924 07:18:58.059048 12795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:58.108279 12795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:58.110715 12795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:58.110910 12795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:58.111164 12795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:58.111500 12795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:58.111563 12795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:58.111599 12795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:58.111627 12795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:58.117417 12795 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:37103
I20260924 07:18:58.117458 12942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:37103 every 8 connection(s)
I20260924 07:18:58.117892 12795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:58.120204 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 12795
I20260924 07:18:58.120507 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:34847
--local_ip_for_outbound_sockets=127.3.246.66
--tserver_master_addrs=127.3.246.126:35777
--webserver_port=36659
--webserver_interface=127.3.246.66
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.246.66
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:18:58.127686 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:58.118682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:58435)
0924 07:18:58.118887 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:58.118937 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:58.119560 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:58.119952 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:58.119956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:58.120770 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:58.120959 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.120964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.122397 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.122399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.123735 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.123738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.124374 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.124375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.124534 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.125068 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:58.125078 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:58.127540 (+ 2462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
W20260924 07:18:58.127946 12946 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:58.118763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:58435 (local address 127.3.246.126:35777)
0924 07:18:58.118991 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:58.118999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:58.119031 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:18:58.119732 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:58.119737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:58.119785 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:58.119898 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:58.121080 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.122301 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.123822 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.124276 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.124354 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.127735 (+ 3381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:58435: BlockingRecv error: recv got EOF from 127.3.246.65:58435 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":125,"threads_started":1}
W20260924 07:18:58.128120 12943 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:35777 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.246.126:35777: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:18:58.132623 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:58.128395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:48233)
0924 07:18:58.128432 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:58.128444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:58.128506 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:58.128631 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:58.128632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:58.128756 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:58.128800 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.128801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.129426 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.129428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.129986 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.129988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.130366 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.130367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.130438 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.130904 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:58.130909 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:58.132554 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 07:18:58.133045 12946 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:58.128448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:48233 (local address 127.3.246.126:35777)
0924 07:18:58.128509 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:58.128511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:58.128520 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:58.128563 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:58.128564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:58.128576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:58.128621 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:58.128877 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.129347 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.130082 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.130285 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.130358 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.132638 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:48233: BlockingRecv error: recv got EOF from 127.3.246.65:48233 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:18:58.137835 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:58.132967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:59383)
0924 07:18:58.133165 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:58.133176 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:58.133231 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:58.133365 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:58.133366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:58.133503 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:58.133534 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.133535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.134213 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.134214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.135013 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.135015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.135390 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.135391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.135446 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.135842 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:58.135847 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:58.137769 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182}
W20260924 07:18:58.137955 12946 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:58.133036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:59383 (local address 127.3.246.126:35777)
0924 07:18:58.133161 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:58.133163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:58.133233 (+ 70us) server_negotiation.cc:407] Connection header received
0924 07:18:58.133281 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:58.133286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:58.133297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:58.133344 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:58.133596 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.134136 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.135109 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.135302 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.135373 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.137869 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:59383: BlockingRecv error: recv got EOF from 127.3.246.65:59383 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":87,"threads_started":1}
W20260924 07:18:58.137982 12943 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:18:58.204703 12947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:58.204911 12947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:58.204936 12947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:58.206130 12947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:58.206168 12947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:58.206195 12947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:18:58.207547 12947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.66.keytab
--principal=oryx/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:34847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=36659
--tserver_master_addrs=127.3.246.126:35777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:58.207732 12947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:58.207908 12947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:58.212882 12956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:58.212896 12953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:58.212875 12954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:58.213157 12947 server_base.cc:1034] running on GCE node
I20260924 07:18:58.213370 12947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:58.213554 12947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:58.214733 12947 hybrid_clock.cc:648] HybridClock initialized: now 1790234338214710 us; error 36 us; skew 500 ppm
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234338, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:58.217348 12947 init.cc:377] Logged in from keytab as oryx/127.3.246.66@KRBTEST.COM (short username oryx)
I20260924 07:18:58.217847 12947 webserver.cc:460] Webserver started at http://127.3.246.66:36659/ using document root <none> and password file <none>
I20260924 07:18:58.218075 12947 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:58.218155 12947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:58.219671 12947 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:18:58.220315 12963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:58.220487 12947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:58.220611 12947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "c3840bf65437419eb7eb736601dc219c"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "4e96790a9d58be7ab5dbeea286a71f4e"
server_key_iv: "21fae5c4e8b143cd435fff49f3177aa2"
server_key_version: "encryptionkey@0"
I20260924 07:18:58.220979 12947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:58.230892 12947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:58.233152 12947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:58.233315 12947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:58.233538 12947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:58.233891 12947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:58.233944 12947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:58.233984 12947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:58.234010 12947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:58.239878 12947 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:34847
I20260924 07:18:58.239928 13076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:34847 every 8 connection(s)
I20260924 07:18:58.240298 12947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:58.245846 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 12947
I20260924 07:18:58.246156 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:33873
--local_ip_for_outbound_sockets=127.3.246.67
--tserver_master_addrs=127.3.246.126:35777
--webserver_port=42997
--webserver_interface=127.3.246.67
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.246.67
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:18:58.248052 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:58.241053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:50291)
0924 07:18:58.241203 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:58.241246 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:58.241838 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:58.242097 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:58.242101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:58.242659 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:58.242846 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.242851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.243556 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.243557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.244359 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.244362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.244767 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.244768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.244901 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.245470 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:58.245478 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:58.247915 (+ 2437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
W20260924 07:18:58.248116 12946 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:58.241095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:50291 (local address 127.3.246.126:35777)
0924 07:18:58.241139 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:58.241141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:58.241341 (+ 200us) server_negotiation.cc:407] Connection header received
0924 07:18:58.241946 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:58.241947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:58.241965 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:58.242028 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:58.242951 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.243457 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.244454 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.244666 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.244754 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.248039 (+ 3285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:50291: BlockingRecv error: recv got EOF from 127.3.246.66:50291 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 07:18:58.248353 13077 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:35777 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.246.126:35777: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:18:58.252784 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:58.248557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:49679)
0924 07:18:58.248608 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:58.248619 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:58.248668 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:58.248806 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:58.248807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:58.248929 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:58.248970 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.248971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.249611 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.249612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.250193 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.250195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.250561 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.250562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.250636 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.251054 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:58.251060 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:58.252703 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:18:58.252836 12946 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:58.248615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:49679 (local address 127.3.246.126:35777)
0924 07:18:58.248665 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:58.248668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:58.248677 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:58.248724 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:58.248725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:58.248736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:58.248803 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:58.249055 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.249528 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.250295 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.250488 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.250563 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.252772 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:49679: BlockingRecv error: recv got EOF from 127.3.246.66:49679 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:18:58.257443 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:58.253066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:49153)
0924 07:18:58.253167 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:58.253179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:58.253226 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:58.253350 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:58.253351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:58.253473 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:58.253499 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.253500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.254236 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.254237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.254739 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.254741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.255130 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.255131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.255201 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.255612 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:58.255618 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:58.257377 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260924 07:18:58.257596 12946 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:58.253110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:49153 (local address 127.3.246.126:35777)
0924 07:18:58.253171 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:58.253174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:58.253211 (+ 37us) server_negotiation.cc:407] Connection header received
0924 07:18:58.253285 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:58.253286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:58.253295 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:58.253338 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:58.253583 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.254160 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.254840 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.255058 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.255128 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.257456 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:49153: BlockingRecv error: recv got EOF from 127.3.246.66:49153 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 07:18:58.257617 13077 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:18:58.328504 13080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:58.328703 13080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:58.328725 13080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:58.329963 13080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:58.330003 13080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:58.330030 13080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:18:58.331516 13080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/krb5kdc/oryx_127.3.246.67.keytab
--principal=oryx/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:33873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=42997
--tserver_master_addrs=127.3.246.126:35777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:18:58.331748 13080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:58.332001 13080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:58.337080 13086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:58.337280 13080 server_base.cc:1034] running on GCE node
W20260924 07:18:58.337069 13088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:58.337069 13085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:58.337571 13080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:58.337879 13080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:58.339020 13080 hybrid_clock.cc:648] HybridClock initialized: now 1790234338339008 us; error 28 us; skew 500 ppm
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234338, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:58.341346 13080 init.cc:377] Logged in from keytab as oryx/127.3.246.67@KRBTEST.COM (short username oryx)
I20260924 07:18:58.341866 13080 webserver.cc:460] Webserver started at http://127.3.246.67:42997/ using document root <none> and password file <none>
I20260924 07:18:58.342087 13080 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:58.342165 13080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:58.343547 13080 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:58.344836 13095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:58.345042 13080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:18:58.345142 13080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "0f71a7e1d2894dc6839ff6fae2248137"
format_stamp: "Formatted at 2026-09-24 07:18:57 on dist-test-slave-rc1w"
server_key: "95889eae4069f3a61339d6f47077deed"
server_key_iv: "9d768a5636d886bec2666e187455551d"
server_key_version: "encryptionkey@0"
I20260924 07:18:58.345496 13080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:58.360219 13080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:58.362790 13080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:58.362960 13080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:58.363207 13080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:58.363517 13080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:58.363574 13080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:58.363616 13080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:58.363641 13080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:58.369491 13080 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:33873
I20260924 07:18:58.369532 13208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:33873 every 8 connection(s)
I20260924 07:18:58.369918 13080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:58.371289 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 13080
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:18:58.378158 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:58.370710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:48979)
0924 07:18:58.370851 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:58.370899 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:58.371472 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:58.371732 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:58.371737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:58.372550 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:58.372839 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.372846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.373548 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.373549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.374560 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.374564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.375008 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.375009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.375147 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.375618 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:58.375626 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:58.378009 (+ 2383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
W20260924 07:18:58.378182 12946 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:58.370744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:48979 (local address 127.3.246.126:35777)
0924 07:18:58.370793 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:58.370796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:58.370988 (+ 192us) server_negotiation.cc:407] Connection header received
0924 07:18:58.371582 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:58.371583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:58.371596 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:58.371659 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:58.372936 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.373459 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.374635 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.374918 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.374999 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.378108 (+ 3109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:48979: BlockingRecv error: recv got EOF from 127.3.246.67:48979 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 07:18:58.378403 13209 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:35777 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.246.126:35777: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
I20260924 07:18:58.379745 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:58.371776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.3.246.126:35777)
0924 07:18:58.371826 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:58.371828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:58.371866 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:18:58.371948 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:58.371948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:58.371957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:58.372014 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:58.372804 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.373311 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.374214 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.374511 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.374588 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.376093 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:58.376098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:58.376098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:58.376112 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:58.378196 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:58.378614 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:58.378618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:58.378618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:58.378757 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:58.378981 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:58.378982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:58.378983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:58.379293 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:58.379389 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:58.379548 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 07:18:58.379633 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:18:58.383131 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:58.378657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:46225)
0924 07:18:58.378711 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:58.378721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:58.378751 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:58.378890 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:58.378891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:58.379045 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:58.379096 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.379097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.379889 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.379890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.380806 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.380808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.381102 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.381103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.381150 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.381476 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:58.381480 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:58.383087 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 07:18:58.383262 12946 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:58.378697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:46225 (local address 127.3.246.126:35777)
0924 07:18:58.378731 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:58.378733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:58.378753 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:18:58.378820 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:58.378821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:58.378830 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:58.378887 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:58.379170 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.379814 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.380867 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.381038 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.381090 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.383185 (+ 2095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:46225: BlockingRecv error: recv got EOF from 127.3.246.67:46225 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:58 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:18:58.387509 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:58.383409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:58911)
0924 07:18:58.383517 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:58.383528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:58.383599 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:58.383737 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:58.383738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:58.383858 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:58.383882 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.383883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.384495 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.384496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.384990 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:58.384992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.385351 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.385352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:58.385410 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.385829 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:58.385833 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:58.387434 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260924 07:18:58.387567 12946 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:58.383458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:58911 (local address 127.3.246.126:35777)
0924 07:18:58.383510 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:58.383511 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:58.383584 (+ 73us) server_negotiation.cc:407] Connection header received
0924 07:18:58.383656 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:58.383657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:58.383666 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:58.383734 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:58.383955 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.384408 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.385075 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:58.385271 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:58.385341 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:58.387497 (+ 2156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:58911: BlockingRecv error: recv got EOF from 127.3.246.67:58911 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 07:18:58.387686 13209 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 07:18:59 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:59 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:18:59.143210 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:59.138354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:57039)
0924 07:18:59.138517 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:59.138528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:59.138584 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:59.138709 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:59.138709 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:59.138863 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:59.138910 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:59.138911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:59.139544 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:59.139545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:59.140079 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:59.140081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:59.140415 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:59.140416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:59.140455 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:59.140966 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:59.140972 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:59.143144 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":103,"threads_started":1}
W20260924 07:18:59.143322 13214 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:59.138462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:57039 (local address 127.3.246.126:35777)
0924 07:18:59.138589 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:59.138591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:59.138600 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:18:59.138625 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:59.138625 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:59.138635 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:59.138694 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:59.138982 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:59.139481 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:59.140154 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:59.140347 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:59.140402 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:59.143240 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:57039: BlockingRecv error: recv got EOF from 127.3.246.65:57039 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
Sep 24 07:18:59 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:59 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:18:59.263028 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:59.258048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:35357)
0924 07:18:59.258213 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:59.258225 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:59.258309 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:59.258490 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:59.258491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:59.258663 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:59.258709 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:59.258710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:59.259572 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:59.259574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:59.260213 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:59.260215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:59.260542 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:59.260543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:59.260582 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:59.260977 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:59.260981 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:59.262961 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
W20260924 07:18:59.263150 13214 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:59.258129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:35357 (local address 127.3.246.126:35777)
0924 07:18:59.258204 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:59.258207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:59.258290 (+ 83us) server_negotiation.cc:407] Connection header received
0924 07:18:59.258388 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:59.258389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:59.258401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:59.258470 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:59.258788 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:59.259465 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:59.260288 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:59.260476 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:59.260528 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:59.263052 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:35357: BlockingRecv error: recv got EOF from 127.3.246.66:35357 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 07:18:59 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:18:59 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:18:59.393021 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:59.388086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:58807)
0924 07:18:59.388291 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:59.388302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:59.388392 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:59.388554 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:59.388555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:59.388743 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:59.388790 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:59.388791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:59.389641 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:59.389642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:59.390239 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:59.390241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:59.390638 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:59.390639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:59.390707 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:59.391200 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:59.391205 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:59.392955 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
W20260924 07:18:59.393091 13214 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:59.388194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:58807 (local address 127.3.246.126:35777)
0924 07:18:59.388269 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:59.388271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:59.388372 (+ 101us) server_negotiation.cc:407] Connection header received
0924 07:18:59.388458 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:59.388459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:59.388472 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:59.388549 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:59.388873 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:59.389576 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:59.390331 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:59.390546 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:59.390635 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:59.393045 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:58807: BlockingRecv error: recv got EOF from 127.3.246.67:58807 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 07:19:00 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:00 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:00.148890 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:00.143734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:46195)
0924 07:19:00.143934 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:00.143946 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:00.144008 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:00.144218 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:00.144219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:00.144389 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:00.144436 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:00.144437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:00.145152 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:00.145152 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:00.145820 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:00.145823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:00.146302 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:00.146304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:00.146375 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:00.146886 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:00.146896 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:00.148835 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":107,"threads_started":1}
W20260924 07:19:00.149020 13218 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:00.143854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:46195 (local address 127.3.246.126:35777)
0924 07:19:00.144004 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:00.144006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:00.144018 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:19:00.144068 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:00.144068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:00.144080 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:00.144140 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:00.144518 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:00.145076 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:00.145906 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:00.146179 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:00.146270 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:00.148929 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:46195: BlockingRecv error: recv got EOF from 127.3.246.65:46195 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
Sep 24 07:19:00 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:00 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:00.268070 13219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:00.263496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:38303)
0924 07:19:00.263646 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:00.263660 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:00.263714 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:00.263843 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:00.263844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:00.264004 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:00.264054 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:00.264056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:00.264690 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:00.264691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:00.265241 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:00.265243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:00.265580 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:00.265581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:00.265625 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:00.266266 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:00.266272 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:00.268011 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
W20260924 07:19:00.268260 13218 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:00.263588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:38303 (local address 127.3.246.126:35777)
0924 07:19:00.263658 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:00.263659 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:00.263719 (+ 60us) server_negotiation.cc:407] Connection header received
0924 07:19:00.263766 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:00.263766 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:00.263777 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:00.263822 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:00.264117 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:00.264607 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:00.265324 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:00.265501 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:00.265551 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:00.268183 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:38303: BlockingRecv error: recv got EOF from 127.3.246.66:38303 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 07:19:00 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:00 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:00.398427 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:00.393543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:34803)
0924 07:19:00.393734 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:00.393749 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:00.393810 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:00.394004 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:00.394005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:00.394182 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:00.394235 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:00.394237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:00.394869 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:00.394869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:00.395416 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:00.395417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:00.395866 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:00.395867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:00.395949 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:00.396481 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:00.396485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:00.398368 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":60,"threads_started":1}
W20260924 07:19:00.398530 13218 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:00.393754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:34803 (local address 127.3.246.126:35777)
0924 07:19:00.393815 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:00.393817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:00.393883 (+ 66us) server_negotiation.cc:407] Connection header received
0924 07:19:00.393912 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:00.393913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:00.393926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:00.393987 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:00.394301 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:00.394791 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:00.395490 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:00.395768 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:00.395852 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:00.398461 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:34803: BlockingRecv error: recv got EOF from 127.3.246.67:34803 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 07:19:01 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:01 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:01.154235 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:01.149370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:41091)
0924 07:19:01.149531 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:01.149544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:01.149612 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:01.149923 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:01.149924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:01.150087 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:01.150137 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:01.150139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:01.150789 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:01.150790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:01.151298 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:01.151299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:01.151622 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:01.151623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:01.151662 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:01.152171 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:01.152179 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:01.154116 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
W20260924 07:19:01.154264 13222 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:01.149434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:41091 (local address 127.3.246.126:35777)
0924 07:19:01.149588 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:01.149590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:01.149598 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:19:01.149658 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:01.149659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:01.149839 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:01.149905 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:01.150206 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:01.150725 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:01.151379 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:01.151561 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:01.151610 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:01.154197 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:41091: BlockingRecv error: recv got EOF from 127.3.246.65:41091 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
Sep 24 07:19:01 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:01 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:01.273113 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:01.268674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:34351)
0924 07:19:01.268824 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:01.268842 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:01.268897 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:01.269025 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:01.269026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:01.269182 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:01.269231 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:01.269232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:01.269880 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:01.269881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:01.270407 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:01.270408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:01.270715 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:01.270716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:01.270757 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:01.271294 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:01.271298 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:01.273055 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
W20260924 07:19:01.273277 13222 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:01.268757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:34351 (local address 127.3.246.126:35777)
0924 07:19:01.268825 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:01.268827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:01.268887 (+ 60us) server_negotiation.cc:407] Connection header received
0924 07:19:01.268942 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:01.268943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:01.268954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:01.269005 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:01.269294 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:01.269817 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:01.270474 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:01.270652 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:01.270698 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:01.273167 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:34351: BlockingRecv error: recv got EOF from 127.3.246.66:34351 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 07:19:01 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:01 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:01.403861 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:01.398989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:59883)
0924 07:19:01.399199 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:01.399213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:01.399312 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:01.399491 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:01.399492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:01.399702 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:01.399766 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:01.399768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:01.400456 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:01.400457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:01.401005 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:01.401007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:01.401365 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:01.401366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:01.401432 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:01.402000 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:01.402005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:01.403802 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":120,"threads_started":1}
W20260924 07:19:01.404002 13222 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:01.399102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:59883 (local address 127.3.246.126:35777)
0924 07:19:01.399203 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:01.399206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:01.399292 (+ 86us) server_negotiation.cc:407] Connection header received
0924 07:19:01.399399 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:01.399399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:01.399413 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:01.399484 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:01.399853 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:01.400360 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:01.401092 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:01.401290 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:01.401366 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:01.403911 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:59883: BlockingRecv error: recv got EOF from 127.3.246.67:59883 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 07:19:02 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:02 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:02.168578 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:02.154813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:43431)
0924 07:19:02.154982 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:02.154994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:02.155054 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:02.155254 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:02.155255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:02.155450 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:02.155501 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:02.155502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:02.156317 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:02.156319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:02.157004 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:02.157007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:02.157421 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:02.157422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:02.157460 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:02.166499 (+ 9039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:02.166504 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:02.168519 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
W20260924 07:19:02.168759 13226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:02.154893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:43431 (local address 127.3.246.126:35777)
0924 07:19:02.155033 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:02.155035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:02.155046 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:19:02.155106 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:02.155107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:02.155118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:02.155204 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:02.155577 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:02.156202 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:02.157099 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:02.157350 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:02.157413 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:02.168668 (+ 11255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:43431: BlockingRecv error: recv got EOF from 127.3.246.65:43431 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
Sep 24 07:19:02 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:02 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:02.278537 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:02.273657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:44503)
0924 07:19:02.273825 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:02.273836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:02.273895 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:02.274052 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:02.274053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:02.274233 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:02.274291 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:02.274293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:02.274953 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:02.274954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:02.275633 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:02.275636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:02.276044 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:02.276045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:02.276115 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:02.276623 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:02.276629 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:02.278477 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":110,"threads_started":1}
W20260924 07:19:02.278721 13226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:02.273824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:44503 (local address 127.3.246.126:35777)
0924 07:19:02.273893 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:02.273895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:02.273907 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:19:02.273950 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:02.273950 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:02.273961 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:02.274003 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:02.274371 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:02.274865 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:02.275731 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:02.275969 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:02.276038 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:02.278648 (+ 2610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:44503: BlockingRecv error: recv got EOF from 127.3.246.66:44503 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 07:19:02 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:02 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:02.409157 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:02.404412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:53353)
0924 07:19:02.404636 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:02.404649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:02.404722 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:02.404883 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:02.404883 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:02.405073 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:02.405128 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:02.405130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:02.405885 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:02.405886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:02.406425 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:02.406427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:02.406789 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:02.406790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:02.406848 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:02.407351 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:02.407356 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:02.409057 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":123,"threads_started":1}
W20260924 07:19:02.409195 13226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:02.404509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:53353 (local address 127.3.246.126:35777)
0924 07:19:02.404618 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:02.404621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:02.404702 (+ 81us) server_negotiation.cc:407] Connection header received
0924 07:19:02.404780 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:02.404781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:02.404801 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:02.404884 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:02.405212 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:02.405802 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:02.406512 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:02.406711 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:02.406790 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:02.409136 (+ 2346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:53353: BlockingRecv error: recv got EOF from 127.3.246.67:53353 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 07:19:03 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:03 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:03.173477 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:03.169032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:56143)
0924 07:19:03.169203 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:03.169213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:03.169266 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:03.169384 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:03.169384 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:03.169548 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:03.169595 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:03.169597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:03.170277 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:03.170278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:03.170833 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:03.170835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:03.171178 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:03.171179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:03.171218 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:03.171640 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:03.171644 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:03.173381 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":104,"threads_started":1}
W20260924 07:19:03.173532 13230 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:03.169137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:56143 (local address 127.3.246.126:35777)
0924 07:19:03.169274 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:03.169276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:03.169286 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:19:03.169309 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:03.169310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:03.169321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:03.169379 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:03.169687 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:03.170209 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:03.170915 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:03.171109 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:03.171164 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:03.173458 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:56143: BlockingRecv error: recv got EOF from 127.3.246.65:56143 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
Sep 24 07:19:03 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:03 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:03.284050 13231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:03.279094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:57977)
0924 07:19:03.279241 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:03.279251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:03.279299 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:03.279483 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:03.279484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:03.279649 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:03.279697 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:03.279699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:03.280351 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:03.280352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:03.280877 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:03.280878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:03.281259 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:03.281260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:03.281300 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:03.281961 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:03.281966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:03.283992 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
W20260924 07:19:03.284161 13230 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:03.279274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:57977 (local address 127.3.246.126:35777)
0924 07:19:03.279356 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:03.279358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:03.279365 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:19:03.279391 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:03.279392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:03.279403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:03.279462 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:03.279777 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:03.280278 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:03.280964 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:03.281176 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:03.281232 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:03.284086 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:57977: BlockingRecv error: recv got EOF from 127.3.246.66:57977 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 07:19:03 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:03 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:03.414635 13232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:03.409732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:36375)
0924 07:19:03.409990 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:03.410002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:03.410058 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:03.410190 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:03.410191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:03.410354 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:03.410403 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:03.410404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:03.411062 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:03.411063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:03.411583 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:03.411584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:03.411937 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:03.411938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:03.411990 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:03.412496 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:03.412501 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:03.414509 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":194,"threads_started":1}
W20260924 07:19:03.414660 13230 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:03.409897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:36375 (local address 127.3.246.126:35777)
0924 07:19:03.409945 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:03.409947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:03.410064 (+ 117us) server_negotiation.cc:407] Connection header received
0924 07:19:03.410103 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:03.410103 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:03.410115 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:03.410171 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:03.410484 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:03.410985 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:03.411661 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:03.411844 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:03.411899 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:03.414612 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:36375: BlockingRecv error: recv got EOF from 127.3.246.67:36375 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 07:19:04 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:04 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:04.179059 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:04.174012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:42181)
0924 07:19:04.174159 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:04.174179 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:04.174230 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:04.174428 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:04.174429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:04.174601 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:04.174648 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:04.174650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:04.175456 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:04.175457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:04.176033 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:04.176035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:04.176441 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:04.176442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:04.176492 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:04.176989 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:04.176994 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:04.178994 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
W20260924 07:19:04.179147 13234 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:04.174108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:42181 (local address 127.3.246.126:35777)
0924 07:19:04.174288 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:04.174290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:04.174301 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:19:04.174325 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:04.174326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:04.174338 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:04.174414 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:04.174733 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:04.175303 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:04.176132 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:04.176350 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:04.176419 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:04.179091 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:42181: BlockingRecv error: recv got EOF from 127.3.246.65:42181 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
Sep 24 07:19:04 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:04 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:04.289417 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:04.284620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:54861)
0924 07:19:04.284809 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:04.284841 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:04.284929 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:04.285106 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:04.285108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:04.285296 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:04.285345 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:04.285347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:04.286136 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:04.286138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:04.286669 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:04.286671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:04.287083 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:04.287084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:04.287138 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:04.287561 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:04.287566 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:04.289357 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":114,"threads_started":1}
W20260924 07:19:04.289529 13234 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:04.284716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:54861 (local address 127.3.246.126:35777)
0924 07:19:04.284794 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:04.284797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:04.284910 (+ 113us) server_negotiation.cc:407] Connection header received
0924 07:19:04.285018 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:04.285019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:04.285031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:04.285098 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:04.285440 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:04.286055 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:04.286761 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:04.286976 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:04.287056 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:04.289453 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:54861: BlockingRecv error: recv got EOF from 127.3.246.66:54861 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 07:19:04 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:04 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:04.419873 13236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:04.415212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:40251)
0924 07:19:04.415377 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:04.415389 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:04.415452 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:04.415603 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:04.415605 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:04.415807 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:04.415856 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:04.415857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:04.416501 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:04.416501 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:04.417018 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:04.417019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:04.417335 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:04.417335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:04.417373 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:04.417852 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:04.417857 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:04.419748 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":74,"threads_started":1}
W20260924 07:19:04.419953 13234 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:04.415287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:40251 (local address 127.3.246.126:35777)
0924 07:19:04.415393 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:04.415397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:04.415436 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:19:04.415505 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:04.415506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:04.415519 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:04.415562 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:04.415932 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:04.416433 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:04.417087 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:04.417268 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:04.417317 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:04.419881 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:40251: BlockingRecv error: recv got EOF from 127.3.246.67:40251 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 07:19:05 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:05 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:05.184345 13237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:05.179623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:56715)
0924 07:19:05.179764 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:05.179778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:05.179833 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:05.180000 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:05.180001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:05.180161 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:05.180211 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:05.180213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:05.180876 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:05.180877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:05.181404 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:05.181405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:05.181782 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:05.181783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:05.181827 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:05.182253 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:05.182257 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:05.184285 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
W20260924 07:19:05.184494 13238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:05.179691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:56715 (local address 127.3.246.126:35777)
0924 07:19:05.179819 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:05.179822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:05.179832 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:19:05.179876 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:05.179876 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:05.179887 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:05.179949 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:05.180282 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:05.180805 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:05.181475 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:05.181657 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:05.181723 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:05.184397 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:56715: BlockingRecv error: recv got EOF from 127.3.246.65:56715 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
Sep 24 07:19:05 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:05 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:05.294528 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:05.289967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:59249)
0924 07:19:05.290131 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:05.290143 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:05.290212 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:05.290352 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:05.290353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:05.290515 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:05.290567 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:05.290568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:05.291214 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:05.291215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:05.291734 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:05.291735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:05.292057 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:05.292057 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:05.292095 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:05.292550 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:05.292555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:05.294435 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":103,"threads_started":1}
W20260924 07:19:05.294688 13238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:05.290076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:59249 (local address 127.3.246.126:35777)
0924 07:19:05.290148 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:05.290150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:05.290213 (+ 63us) server_negotiation.cc:407] Connection header received
0924 07:19:05.290259 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:05.290260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:05.290270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:05.290325 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:05.290634 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:05.291133 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:05.291807 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:05.291992 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:05.292044 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:05.294565 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:59249: BlockingRecv error: recv got EOF from 127.3.246.66:59249 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 07:19:05 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:05 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:05.424940 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:05.420341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:43559)
0924 07:19:05.420503 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:05.420514 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:05.420572 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:05.420727 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:05.420728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:05.420882 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:05.420932 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:05.420934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:05.421553 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:05.421554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:05.422090 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:05.422091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:05.422419 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:05.422420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:05.422460 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:05.423058 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:05.423063 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:05.424881 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
W20260924 07:19:05.425069 13238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:05.420405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:43559 (local address 127.3.246.126:35777)
0924 07:19:05.420471 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:05.420473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:05.420616 (+ 143us) server_negotiation.cc:407] Connection header received
0924 07:19:05.420643 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:05.420645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:05.420665 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:05.420706 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:05.420998 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:05.421487 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:05.422162 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:05.422354 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:05.422406 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:05.424969 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:43559: BlockingRecv error: recv got EOF from 127.3.246.67:43559 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 07:19:06 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:06 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:06.189874 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:06.184839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:52131)
0924 07:19:06.184998 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:06.185018 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:06.185080 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:06.185252 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:06.185253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:06.185431 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:06.185485 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:06.185487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:06.186225 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:06.186226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:06.186789 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:06.186791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:06.187194 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:06.187195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:06.187244 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:06.187849 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:06.187854 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:06.189806 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":57,"threads_started":1}
W20260924 07:19:06.189944 13242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:06.184988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:52131 (local address 127.3.246.126:35777)
0924 07:19:06.185117 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:06.185119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:06.185127 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:19:06.185158 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:06.185158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:06.185169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:06.185228 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:06.185557 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:06.186137 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:06.186879 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:06.187119 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:06.187176 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:06.189889 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:52131: BlockingRecv error: recv got EOF from 127.3.246.65:52131 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
Sep 24 07:19:06 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:06 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:06.300036 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:06.295002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:40047)
0924 07:19:06.295221 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:06.295237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:06.295355 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:06.295518 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:06.295519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:06.295699 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:06.295747 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:06.295749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:06.296602 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:06.296603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:06.297141 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:06.297143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:06.297540 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:06.297541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:06.297605 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:06.298186 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:06.298192 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:06.299967 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":101,"threads_started":1}
W20260924 07:19:06.300163 13242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:06.295113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:40047 (local address 127.3.246.126:35777)
0924 07:19:06.295189 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:06.295192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:06.295299 (+ 107us) server_negotiation.cc:407] Connection header received
0924 07:19:06.295451 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:06.295452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:06.295464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:06.295513 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:06.295849 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:06.296505 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:06.297232 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:06.297447 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:06.297536 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:06.300064 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:40047: BlockingRecv error: recv got EOF from 127.3.246.66:40047 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 07:19:06 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:06 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:06.430043 13244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:06.425390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:51771)
0924 07:19:06.425545 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:06.425556 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:06.425611 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:06.425821 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:06.425822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:06.426029 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:06.426078 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:06.426080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:06.426711 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:06.426712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:06.427226 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:06.427227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:06.427594 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:06.427595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:06.427637 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:06.428080 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:06.428085 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:06.429984 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
W20260924 07:19:06.430274 13242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:06.425495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:51771 (local address 127.3.246.126:35777)
0924 07:19:06.425566 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:06.425568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:06.425609 (+ 41us) server_negotiation.cc:407] Connection header received
0924 07:19:06.425654 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:06.425654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:06.425729 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:06.425785 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:06.426149 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:06.426643 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:06.427338 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:06.427525 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:06.427588 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:06.430214 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:51771: BlockingRecv error: recv got EOF from 127.3.246.67:51771 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 07:19:07 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:07 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:07.205338 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:07.190467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:45055)
0924 07:19:07.190668 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:07.190679 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:07.190734 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:07.190954 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:07.190955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:07.191170 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:07.191221 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:07.191223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:07.191956 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:07.191957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:07.192474 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:07.192475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:07.192855 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:07.192856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:07.192895 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:07.203285 (+ 10390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:07.203291 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:07.205238 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":112,"threads_started":1}
W20260924 07:19:07.205403 13246 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:07.190537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:45055 (local address 127.3.246.126:35777)
0924 07:19:07.190710 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:07.190712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:07.190728 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:19:07.190827 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:07.190828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:07.190844 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:07.190943 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:07.191323 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:07.191876 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:07.192572 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:07.192779 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:07.192845 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:07.205329 (+ 12484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:45055: BlockingRecv error: recv got EOF from 127.3.246.65:45055 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
Sep 24 07:19:07 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:07 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:07.304837 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:07.300458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:53459)
0924 07:19:07.300584 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:07.300593 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:07.300640 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:07.300764 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:07.300765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:07.300938 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:07.300988 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:07.300989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:07.301620 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:07.301621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:07.302179 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:07.302180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:07.302498 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:07.302499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:07.302536 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:07.302957 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:07.302961 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:07.304772 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":53,"threads_started":1}
W20260924 07:19:07.304940 13246 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:07.300535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:53459 (local address 127.3.246.126:35777)
0924 07:19:07.300593 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:07.300595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:07.300629 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:19:07.300695 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:07.300695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:07.300705 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:07.300744 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:07.301053 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:07.301555 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:07.302249 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:07.302437 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:07.302489 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:07.304873 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:53459: BlockingRecv error: recv got EOF from 127.3.246.66:53459 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 07:19:07 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:07 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:07.435175 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:07.430526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:34927)
0924 07:19:07.430653 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:07.430673 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:07.430741 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:07.430881 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:07.430882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:07.431059 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:07.431105 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:07.431107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:07.431728 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:07.431730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:07.432254 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:07.432255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:07.432631 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:07.432632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:07.432682 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:07.433141 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:07.433145 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:07.435083 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":53,"threads_started":1}
W20260924 07:19:07.435220 13246 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:07.430619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:34927 (local address 127.3.246.126:35777)
0924 07:19:07.430667 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:07.430669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:07.430741 (+ 72us) server_negotiation.cc:407] Connection header received
0924 07:19:07.430785 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:07.430786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:07.430814 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:07.430860 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:07.431166 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:07.431650 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:07.432330 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:07.432519 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:07.432572 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:07.435171 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:34927: BlockingRecv error: recv got EOF from 127.3.246.67:34927 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 07:19:08 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:08 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:08.210592 13249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:08.205923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:55709)
0924 07:19:08.206123 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:08.206135 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:08.206203 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:08.206357 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:08.206358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:08.206524 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:08.206575 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:08.206576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:08.207247 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:08.207248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:08.207779 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:08.207780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:08.208120 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:08.208121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:08.208163 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:08.208663 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:08.208668 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:08.210530 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":90,"threads_started":1}
W20260924 07:19:08.210737 13250 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:08.206059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:55709 (local address 127.3.246.126:35777)
0924 07:19:08.206187 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:08.206189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:08.206198 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:19:08.206252 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:08.206253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:08.206265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:08.206332 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:08.206646 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:08.207166 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:08.207859 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:08.208046 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:08.208096 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:08.210644 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:55709: BlockingRecv error: recv got EOF from 127.3.246.65:55709 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
Sep 24 07:19:08 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:08 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:08.310424 13251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:08.305410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:38027)
0924 07:19:08.305560 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:08.305576 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:08.305644 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:08.305843 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:08.305844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:08.306009 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:08.306060 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:08.306061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:08.306739 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:08.306740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:08.307320 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:08.307323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:08.307713 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:08.307713 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:08.307752 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:08.308252 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:08.308257 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:08.310316 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
W20260924 07:19:08.310485 13250 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:08.305517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:38027 (local address 127.3.246.126:35777)
0924 07:19:08.305562 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:08.305566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:08.305626 (+ 60us) server_negotiation.cc:407] Connection header received
0924 07:19:08.305746 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:08.305747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:08.305758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:08.305816 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:08.306141 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:08.306666 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:08.307417 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:08.307641 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:08.307699 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:08.310420 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:38027: BlockingRecv error: recv got EOF from 127.3.246.66:38027 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 07:19:08 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:08 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:08.440490 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:08.435813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:53855)
0924 07:19:08.436003 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:08.436016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:08.436078 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:08.436221 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:08.436221 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:08.436434 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:08.436483 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:08.436484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:08.437116 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:08.437117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:08.437637 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:08.437638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:08.438062 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:08.438062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:08.438109 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:08.438549 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:08.438555 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:08.440431 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
W20260924 07:19:08.440634 13250 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:08.435981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:53855 (local address 127.3.246.126:35777)
0924 07:19:08.436047 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:08.436049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:08.436061 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:19:08.436137 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:08.436138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:08.436152 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:08.436200 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:08.436537 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:08.437047 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:08.437768 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:08.437996 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:08.438050 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:08.440552 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:53855: BlockingRecv error: recv got EOF from 127.3.246.67:53855 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 07:19:09 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:09 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:09.216130 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:09.211204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:41783)
0924 07:19:09.211323 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:09.211341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:09.211390 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:09.211553 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:09.211553 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:09.211730 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:09.211776 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:09.211778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:09.212475 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:09.212477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:09.213034 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:09.213036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:09.213444 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:09.213445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:09.213498 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:09.214075 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:09.214079 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:09.216061 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
W20260924 07:19:09.216230 13254 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:09.211277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:41783 (local address 127.3.246.126:35777)
0924 07:19:09.211417 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:09.211420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:09.211435 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:19:09.211458 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:09.211458 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:09.211468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:09.211531 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:09.211858 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:09.212378 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:09.213135 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:09.213352 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:09.213415 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:09.216156 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:41783: BlockingRecv error: recv got EOF from 127.3.246.65:41783 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
Sep 24 07:19:09 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:09 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:09.315920 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:09.310952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:41675)
0924 07:19:09.311141 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:09.311155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:09.311270 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:09.311454 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:09.311455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:09.311654 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:09.311701 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:09.311703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:09.312406 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:09.312407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:09.313004 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:09.313006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:09.313433 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:09.313434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:09.313497 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:09.314046 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:09.314056 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:09.315808 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
W20260924 07:19:09.315956 13254 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:09.311075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:41675 (local address 127.3.246.126:35777)
0924 07:19:09.311161 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:09.311164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:09.311235 (+ 71us) server_negotiation.cc:407] Connection header received
0924 07:19:09.311364 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:09.311364 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:09.311376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:09.311454 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:09.311808 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:09.312320 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:09.313091 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:09.313357 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:09.313436 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:09.315889 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:41675: BlockingRecv error: recv got EOF from 127.3.246.66:41675 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 07:19:09 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:09 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:09.446075 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:09.440943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:33637)
0924 07:19:09.441160 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:09.441174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:09.441272 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:09.441433 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:09.441434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:09.441626 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:09.441723 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:09.441724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:09.442405 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:09.442406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:09.442965 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:09.442967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:09.443336 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:09.443337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:09.443396 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:09.444068 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:09.444074 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:09.446009 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":101,"threads_started":1}
W20260924 07:19:09.446192 13254 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:09.441051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:33637 (local address 127.3.246.126:35777)
0924 07:19:09.441144 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:09.441147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:09.441250 (+ 103us) server_negotiation.cc:407] Connection header received
0924 07:19:09.441331 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:09.441332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:09.441348 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:09.441424 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:09.441811 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:09.442316 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:09.443066 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:09.443263 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:09.443339 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:09.446110 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:33637: BlockingRecv error: recv got EOF from 127.3.246.67:33637 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 07:19:10 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:10 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:10.221361 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.216600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:42695)
0924 07:19:10.216751 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:10.216766 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:10.216827 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:10.216957 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:10.216957 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:10.217121 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:10.217166 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.217168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.217891 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.217892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.218436 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.218438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.218771 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.218772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.218814 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.219353 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:10.219357 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:10.221305 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
W20260924 07:19:10.221487 13258 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:10.216640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:42695 (local address 127.3.246.126:35777)
0924 07:19:10.216759 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.216761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.216830 (+ 69us) server_negotiation.cc:407] Connection header received
0924 07:19:10.216869 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.216869 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.216879 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.216938 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:10.217226 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.217767 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.218508 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.218704 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.218770 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.221396 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:42695: BlockingRecv error: recv got EOF from 127.3.246.65:42695 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
Sep 24 07:19:10 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:10 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:10.321314 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.316469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:50121)
0924 07:19:10.316636 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:10.316646 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:10.316729 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:10.316865 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:10.316866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:10.317031 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:10.317078 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.317080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.317962 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.317963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.318503 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.318505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.318889 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.318890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.318941 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.319393 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:10.319398 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:10.321253 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
W20260924 07:19:10.321455 13258 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:10.316557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:50121 (local address 127.3.246.126:35777)
0924 07:19:10.316626 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.316628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.316726 (+ 98us) server_negotiation.cc:407] Connection header received
0924 07:19:10.316775 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.316776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.316787 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.316844 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:10.317217 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.317879 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.318576 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.318801 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.318854 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.321373 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:50121: BlockingRecv error: recv got EOF from 127.3.246.66:50121 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 07:19:10 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:10 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:10.451128 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.446535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:54823)
0924 07:19:10.446671 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:10.446682 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:10.446736 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:10.446886 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:10.446887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:10.447050 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:10.447099 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.447100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.447719 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.447720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.448244 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.448245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.448578 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.448579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.448676 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.449193 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:10.449197 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:10.451023 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
W20260924 07:19:10.451232 13258 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:10.446618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:54823 (local address 127.3.246.126:35777)
0924 07:19:10.446674 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.446676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.446721 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:19:10.446802 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.446803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.446814 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.446868 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:10.447162 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.447651 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.448312 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.448491 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.448544 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.451130 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:54823: BlockingRecv error: recv got EOF from 127.3.246.67:54823 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 07:19:11 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:11 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:11.226794 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:11.221912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:48829)
0924 07:19:11.222082 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:11.222093 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:11.222154 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:11.222308 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:11.222309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:11.222475 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:11.222522 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:11.222524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:11.223258 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:11.223259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:11.223877 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:11.223880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:11.224259 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:11.224260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:11.224309 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:11.224818 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:11.224823 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:11.226727 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
W20260924 07:19:11.226894 13262 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:11.221996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:48829 (local address 127.3.246.126:35777)
0924 07:19:11.222097 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:11.222100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:11.222197 (+ 97us) server_negotiation.cc:407] Connection header received
0924 07:19:11.222220 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:11.222220 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:11.222232 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:11.222283 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:11.222602 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:11.223187 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:11.223953 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:11.224194 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:11.224247 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:11.226824 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:48829: BlockingRecv error: recv got EOF from 127.3.246.65:48829 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":50,"threads_started":1}
Sep 24 07:19:11 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:11 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:11.326498 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:11.321810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:35419)
0924 07:19:11.321929 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:11.321941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:11.322000 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:11.322146 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:11.322146 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:11.322308 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:11.322354 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:11.322356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:11.323224 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:11.323225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:11.323817 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:11.323820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:11.324213 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:11.324214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:11.324279 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:11.324725 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:11.324730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:11.326429 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
W20260924 07:19:11.326608 13262 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:11.321947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:35419 (local address 127.3.246.126:35777)
0924 07:19:11.322010 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:11.322014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:11.322047 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:19:11.322072 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:11.322072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:11.322083 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:11.322127 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:11.322428 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:11.323139 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:11.323908 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:11.324123 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:11.324210 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:11.326516 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:35419: BlockingRecv error: recv got EOF from 127.3.246.66:35419 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 07:19:11 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:11 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:11.456797 13264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:11.451676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:51723)
0924 07:19:11.451830 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:11.451843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:11.451921 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:11.452104 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:11.452106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:11.452310 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:11.452357 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:11.452359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:11.453044 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:11.453045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:11.453649 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:11.453651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:11.454067 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:11.454067 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:11.454132 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:11.454788 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:11.454797 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:11.456731 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
W20260924 07:19:11.456962 13262 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:11.451748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:51723 (local address 127.3.246.126:35777)
0924 07:19:11.451793 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:11.451796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:11.451914 (+ 118us) server_negotiation.cc:407] Connection header received
0924 07:19:11.452000 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:11.452001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:11.452023 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:11.452094 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:11.452441 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:11.452957 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:11.453772 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:11.453979 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:11.454065 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:11.456882 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:51723: BlockingRecv error: recv got EOF from 127.3.246.67:51723 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 07:19:12 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:12 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:12.246124 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:12.227348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:39673)
0924 07:19:12.227513 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:12.227525 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:12.227588 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:12.227725 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:12.227726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:12.227887 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:12.227937 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:12.227938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:12.228589 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:12.228590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:12.229165 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:12.229167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:12.229487 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:12.229487 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:12.229535 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:12.244283 (+ 14748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:12.244288 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:12.246070 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
W20260924 07:19:12.246258 13266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:12.227469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:39673 (local address 127.3.246.126:35777)
0924 07:19:12.227593 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:12.227595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:12.227604 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:19:12.227640 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:12.227641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:12.227651 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:12.227700 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:12.228003 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:12.228496 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:12.229239 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:12.229421 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:12.229469 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:12.246174 (+ 16705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:39673: BlockingRecv error: recv got EOF from 127.3.246.65:39673 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
Sep 24 07:19:12 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:12 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:12.331879 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:12.326959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:40005)
0924 07:19:12.327143 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:12.327157 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:12.327250 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:12.327425 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:12.327426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:12.327609 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:12.327655 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:12.327657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:12.328345 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:12.328346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:12.328879 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:12.328881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:12.329300 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:12.329302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:12.329397 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:12.329976 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:12.329988 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:12.331819 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":83,"threads_started":1}
W20260924 07:19:12.332059 13266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:12.327057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:40005 (local address 127.3.246.126:35777)
0924 07:19:12.327147 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:12.327150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:12.327230 (+ 80us) server_negotiation.cc:407] Connection header received
0924 07:19:12.327323 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:12.327324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:12.327342 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:12.327406 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:12.327742 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:12.328238 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:12.328962 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:12.329162 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:12.329252 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:12.331953 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:40005: BlockingRecv error: recv got EOF from 127.3.246.66:40005 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 07:19:12 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:12 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.67@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:12.462556 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:12.457352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.67:51633)
0924 07:19:12.457659 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:12.457718 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:12.457802 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:12.457976 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:12.457977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:12.458162 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:12.458206 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:12.458208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:12.458885 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:12.458887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:12.459458 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:12.459460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:12.459960 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:12.459962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:12.460035 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:12.460558 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:12.460563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:12.462487 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":119,"threads_started":1}
W20260924 07:19:12.462648 13266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:12.457465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:51633 (local address 127.3.246.126:35777)
0924 07:19:12.457642 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:12.457645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:12.457781 (+ 136us) server_negotiation.cc:407] Connection header received
0924 07:19:12.457880 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:12.457881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:12.457896 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:12.457965 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:12.458295 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:12.458794 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:12.459543 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:12.459870 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:12.459948 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:12.462581 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.67:51633: BlockingRecv error: recv got EOF from 127.3.246.67:51633 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
Sep 24 07:19:13 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:13 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.65@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:13.251503 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:13.246745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.65:36927)
0924 07:19:13.246885 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:13.246900 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:13.246955 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:13.247097 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:13.247097 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:13.247263 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:13.247316 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:13.247318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.247998 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.247999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:13.248674 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:13.248677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.249048 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.249049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:13.249110 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:13.249563 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:13.249569 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:13.251438 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
W20260924 07:19:13.251605 13270 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:13.246800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:36927 (local address 127.3.246.126:35777)
0924 07:19:13.246927 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:13.246933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:13.246941 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:19:13.247004 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:13.247004 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:13.247015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:13.247085 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:13.247409 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.247916 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.248766 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.248976 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.249030 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:13.251532 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.65:36927: BlockingRecv error: recv got EOF from 127.3.246.65:36927 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
Sep 24 07:19:13 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:19:13 dist-test-slave-rc1w krb5kdc[12212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.246.66@KRBTEST.COM for oryx/127.3.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:19:13.337111 13271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:13.332449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35777 (local address 127.3.246.66:37535)
0924 07:19:13.332608 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:13.332619 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:13.332704 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:13.332872 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:13.332874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:13.333141 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:13.333202 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:13.333205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.333868 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.333869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:13.334409 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:13.334411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.334754 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.334755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:13.334800 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:13.335284 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:13.335288 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:13.337045 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.246.126:35777: Server oryx/127.3.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
W20260924 07:19:13.337204 13270 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:13.332542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:37535 (local address 127.3.246.126:35777)
0924 07:19:13.332595 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:13.332598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:13.332710 (+ 112us) server_negotiation.cc:407] Connection header received
0924 07:19:13.332771 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:13.332772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:13.332786 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:13.332845 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:13.333281 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.333792 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.334493 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.334683 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.334744 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:13.337137 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.246.66:37535: BlockingRecv error: recv got EOF from 127.3.246.66:37535 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 07:19:13.375504 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 12795
I20260924 07:19:13.379638 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 12947
I20260924 07:19:13.383651 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 13080
I20260924 07:19:13.387977 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 12725
2026-09-24T07:19:13Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17521 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T07:19:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:19:13Z Disabled control of system clock
I20260924 07:19:13.415230 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:36045
--webserver_interface=127.3.246.126
--webserver_port=0
--builtin_ntp_servers=127.3.246.84:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:36045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 07:19:13.495852 13275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:13.496064 13275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:13.496083 13275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:13.497200 13275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:19:13.497237 13275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:13.497249 13275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:13.497262 13275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:19:13.497274 13275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:19:13.498548 13275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:36045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:36045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:13.498742 13275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:13.499047 13275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:13.504276 13283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:13.504263 13280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:13.504268 13275 server_base.cc:1034] running on GCE node
W20260924 07:19:13.504231 13281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:13.504755 13275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:13.505062 13275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:13.506224 13275 hybrid_clock.cc:648] HybridClock initialized: now 1790234353506190 us; error 53 us; skew 500 ppm
I20260924 07:19:13.507540 13275 webserver.cc:460] Webserver started at http://127.3.246.126:33019/ using document root <none> and password file <none>
I20260924 07:19:13.507786 13275 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:13.507858 13275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:13.508008 13275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:13.509123 13275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "ad21b466655344e08fe8bbcbc1774147"
format_stamp: "Formatted at 2026-09-24 07:19:13 on dist-test-slave-rc1w"
server_key: "696cf2662fc7749aab5f2698516c0417"
server_key_iv: "3ecee56c8a5c4466bb34d40de232f60a"
server_key_version: "encryptionkey@0"
I20260924 07:19:13.509491 13275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "ad21b466655344e08fe8bbcbc1774147"
format_stamp: "Formatted at 2026-09-24 07:19:13 on dist-test-slave-rc1w"
server_key: "696cf2662fc7749aab5f2698516c0417"
server_key_iv: "3ecee56c8a5c4466bb34d40de232f60a"
server_key_version: "encryptionkey@0"
I20260924 07:19:13.510905 13275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:19:13.511746 13289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:13.511963 13275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:19:13.512068 13275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "ad21b466655344e08fe8bbcbc1774147"
format_stamp: "Formatted at 2026-09-24 07:19:13 on dist-test-slave-rc1w"
server_key: "696cf2662fc7749aab5f2698516c0417"
server_key_iv: "3ecee56c8a5c4466bb34d40de232f60a"
server_key_version: "encryptionkey@0"
I20260924 07:19:13.512151 13275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:13.524871 13275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:13.525187 13275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:13.525339 13275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:13.529316 13275 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:36045
I20260924 07:19:13.529359 13341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:36045 every 8 connection(s)
I20260924 07:19:13.529771 13275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:19:13.530426 13342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:13.532963 13342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147: Bootstrap starting.
I20260924 07:19:13.533504 13342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:13.533813 13342 log.cc:824] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:13.534518 13342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147: No bootstrap required, opened a new log
I20260924 07:19:13.536062 13342 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad21b466655344e08fe8bbcbc1774147" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 36045 } }
I20260924 07:19:13.536199 13342 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:13.536232 13342 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad21b466655344e08fe8bbcbc1774147, State: Initialized, Role: FOLLOWER
I20260924 07:19:13.536334 13342 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad21b466655344e08fe8bbcbc1774147" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 36045 } }
I20260924 07:19:13.536384 13342 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:19:13.536404 13342 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:19:13.536435 13342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:13.536926 13342 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad21b466655344e08fe8bbcbc1774147" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 36045 } }
I20260924 07:19:13.537009 13342 leader_election.cc:302] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad21b466655344e08fe8bbcbc1774147; no voters:
I20260924 07:19:13.537122 13342 leader_election.cc:288] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:19:13.537186 13345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:19:13.537364 13345 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [term 1 LEADER]: Becoming Leader. State: Replica: ad21b466655344e08fe8bbcbc1774147, State: Running, Role: LEADER
I20260924 07:19:13.537451 13342 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:19:13.537559 13345 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad21b466655344e08fe8bbcbc1774147" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 36045 } }
I20260924 07:19:13.538092 13347 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad21b466655344e08fe8bbcbc1774147. Latest consensus state: current_term: 1 leader_uuid: "ad21b466655344e08fe8bbcbc1774147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad21b466655344e08fe8bbcbc1774147" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 36045 } } }
I20260924 07:19:13.538228 13346 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad21b466655344e08fe8bbcbc1774147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad21b466655344e08fe8bbcbc1774147" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 36045 } } }
I20260924 07:19:13.538327 13347 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:13.538335 13346 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147 [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:13.538828 13354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:19:13.539353 13354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:19:13.539587 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 13275
I20260924 07:19:13.539693 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:19:13.539888 4057 external_mini_cluster.cc:1428] Setting key 4346d84c05ed5eb081750cb27b462e3d
I20260924 07:19:13.540939 13354 catalog_manager.cc:1347] Generated new cluster ID: 2b7d1a6d6b674198834605f3b1a3b24c
I20260924 07:19:13.541023 13354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:19:13.545275 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:13.540529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.3.246.126:36045)
0924 07:19:13.540712 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:13.540721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:13.540749 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:19:13.540864 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:13.540872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:13.540942 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:13.541295 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:13.541373 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.542442 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.543281 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.543731 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.543876 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:13.544605 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:13.544609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:13.544609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:19:13.544797 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:13.544830 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:19:13.544841 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:13.544919 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:13.544984 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 07:19:13.545062 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260924 07:19:13.557693 13354 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:19:13.558244 13354 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:19:13.569635 13354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad21b466655344e08fe8bbcbc1774147: Generated new TSK 0
I20260924 07:19:13.569916 13354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:19:13.578253 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:36045
--builtin_ntp_servers=127.3.246.84:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 07:19:13.661770 13366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:13.661998 13366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:13.662029 13366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:13.663172 13366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:13.663239 13366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:13.663285 13366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:19:13.664649 13366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:36045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:13.664907 13366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:13.665208 13366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:13.670286 13372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:13.670234 13374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:13.670234 13371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:13.670660 13366 server_base.cc:1034] running on GCE node
I20260924 07:19:13.670855 13366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:13.671051 13366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:13.672194 13366 hybrid_clock.cc:648] HybridClock initialized: now 1790234353672180 us; error 27 us; skew 500 ppm
I20260924 07:19:13.673429 13366 webserver.cc:460] Webserver started at http://127.3.246.65:39553/ using document root <none> and password file <none>
I20260924 07:19:13.673625 13366 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:13.673708 13366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:13.673842 13366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:13.674882 13366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "39847617d9124454a1d7751bc7f4badf"
format_stamp: "Formatted at 2026-09-24 07:19:13 on dist-test-slave-rc1w"
server_key: "c7148a57190451aa35e10d54b4d0037e"
server_key_iv: "7fa7965b5297a951c95506e7a348b1cc"
server_key_version: "encryptionkey@0"
I20260924 07:19:13.675190 13366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "39847617d9124454a1d7751bc7f4badf"
format_stamp: "Formatted at 2026-09-24 07:19:13 on dist-test-slave-rc1w"
server_key: "c7148a57190451aa35e10d54b4d0037e"
server_key_iv: "7fa7965b5297a951c95506e7a348b1cc"
server_key_version: "encryptionkey@0"
I20260924 07:19:13.676491 13366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:19:13.677179 13380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:13.677480 13366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:19:13.677597 13366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "39847617d9124454a1d7751bc7f4badf"
format_stamp: "Formatted at 2026-09-24 07:19:13 on dist-test-slave-rc1w"
server_key: "c7148a57190451aa35e10d54b4d0037e"
server_key_iv: "7fa7965b5297a951c95506e7a348b1cc"
server_key_version: "encryptionkey@0"
I20260924 07:19:13.677733 13366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:13.690325 13366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:13.690680 13366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:13.690847 13366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:13.691099 13366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:13.691447 13366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:13.691502 13366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:13.691537 13366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:13.691581 13366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:13.698325 13366 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:35825
I20260924 07:19:13.698361 13493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:35825 every 8 connection(s)
I20260924 07:19:13.698742 13366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:19:13.703258 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 13366
I20260924 07:19:13.703326 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:19:13.703538 4057 external_mini_cluster.cc:1428] Setting key ed3ea07d332e7b801fcb277e9efa2954
I20260924 07:19:13.704447 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:13.699623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:38031 (local address 127.3.246.126:36045)
0924 07:19:13.699716 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:13.699718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:13.699970 (+ 252us) server_negotiation.cc:407] Connection header received
0924 07:19:13.700407 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:13.700409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:13.700423 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:13.700497 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:13.701153 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.701705 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.702689 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.702900 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.702959 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:13.704075 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:13.704077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:13.704077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:19:13.704081 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:13.704102 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:19:13.704106 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:13.704162 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:13.704323 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 07:19:13.704358 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:19:13.704643 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:13.699584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:36045 (local address 127.3.246.65:38031)
0924 07:19:13.699797 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:13.699839 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:13.700329 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:13.700551 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:13.700554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:13.700562 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:13.701047 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:13.701051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.701792 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.701793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:13.702598 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:13.702601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.703110 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.703111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:13.703237 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:13.703909 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:19:13.703910 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:13.703950 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:19:13.703979 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:13.704182 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:13.704186 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:13.704189 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:19:13.704257 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:19:13.704387 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":111,"threads_started":1}
I20260924 07:19:13.704838 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:36045
--builtin_ntp_servers=127.3.246.84:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 07:19:13.705140 13494 heartbeater.cc:347] Connected to a master server at 127.3.246.126:36045
I20260924 07:19:13.705237 13494 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:13.705437 13494 heartbeater.cc:511] Master 127.3.246.126:36045 requested a full tablet report, sending...
I20260924 07:19:13.705981 13306 ts_manager.cc:194] Registered new tserver with Master: 39847617d9124454a1d7751bc7f4badf (127.3.246.65:35825)
I20260924 07:19:13.706945 13306 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.246.65:38031
W20260924 07:19:13.785768 13497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:13.785976 13497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:13.785998 13497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:13.787179 13497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:13.787225 13497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:13.787250 13497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:19:13.788605 13497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:36045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:13.788858 13497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:13.789165 13497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:13.794284 13505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:13.794270 13503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:13.794266 13502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:13.794536 13497 server_base.cc:1034] running on GCE node
I20260924 07:19:13.794870 13497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:13.795149 13497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:13.796340 13497 hybrid_clock.cc:648] HybridClock initialized: now 1790234353796328 us; error 29 us; skew 500 ppm
I20260924 07:19:13.797820 13497 webserver.cc:460] Webserver started at http://127.3.246.66:36767/ using document root <none> and password file <none>
I20260924 07:19:13.798067 13497 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:13.798136 13497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:13.798300 13497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:13.799399 13497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "94e58704a52c4792a750aa74ba470b7e"
format_stamp: "Formatted at 2026-09-24 07:19:13 on dist-test-slave-rc1w"
server_key: "79618b75d12af4e4619f97ccb8535ae1"
server_key_iv: "b4eb9bbe131422525761f8e239d40baf"
server_key_version: "encryptionkey@0"
I20260924 07:19:13.799775 13497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "94e58704a52c4792a750aa74ba470b7e"
format_stamp: "Formatted at 2026-09-24 07:19:13 on dist-test-slave-rc1w"
server_key: "79618b75d12af4e4619f97ccb8535ae1"
server_key_iv: "b4eb9bbe131422525761f8e239d40baf"
server_key_version: "encryptionkey@0"
I20260924 07:19:13.801184 13497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:19:13.801980 13511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:13.802186 13497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:19:13.802299 13497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "94e58704a52c4792a750aa74ba470b7e"
format_stamp: "Formatted at 2026-09-24 07:19:13 on dist-test-slave-rc1w"
server_key: "79618b75d12af4e4619f97ccb8535ae1"
server_key_iv: "b4eb9bbe131422525761f8e239d40baf"
server_key_version: "encryptionkey@0"
I20260924 07:19:13.802377 13497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:13.811440 13497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:13.811751 13497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:13.811916 13497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:13.812270 13497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:13.812706 13497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:13.812770 13497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:13.812806 13497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:13.812834 13497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:13.818974 13497 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:44143
I20260924 07:19:13.819012 13624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:44143 every 8 connection(s)
I20260924 07:19:13.819365 13497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:19:13.820022 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 13497
I20260924 07:19:13.820118 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:19:13.820366 4057 external_mini_cluster.cc:1428] Setting key 534ba15ffb00dece4bb5bde6927970cb
I20260924 07:19:13.821434 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:36045
--builtin_ntp_servers=127.3.246.84:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 07:19:13.825467 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:13.820197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:40831 (local address 127.3.246.126:36045)
0924 07:19:13.820252 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:13.820254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:13.820586 (+ 332us) server_negotiation.cc:407] Connection header received
0924 07:19:13.821301 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:13.821302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:13.821313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:13.821364 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:13.822355 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.822890 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.823931 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.824139 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.824218 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:13.825153 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:13.825155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:13.825155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:19:13.825160 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:13.825180 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:19:13.825189 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:13.825244 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:13.825354 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 07:19:13.825387 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 07:19:13.825717 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:13.820163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:36045 (local address 127.3.246.66:40831)
0924 07:19:13.820453 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:13.820498 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:13.821155 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:13.821461 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:13.821465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:13.821477 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:13.822245 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:13.822252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.823011 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.823012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:13.823823 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:13.823825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.824229 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.824230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:13.824371 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:13.824949 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:19:13.824950 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:13.825004 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:19:13.825034 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:13.825259 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:13.825265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:13.825268 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:19:13.825357 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 07:19:13.825461 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":90,"threads_started":1}
I20260924 07:19:13.826433 13625 heartbeater.cc:347] Connected to a master server at 127.3.246.126:36045
I20260924 07:19:13.826521 13625 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:13.826746 13625 heartbeater.cc:511] Master 127.3.246.126:36045 requested a full tablet report, sending...
I20260924 07:19:13.827159 13306 ts_manager.cc:194] Registered new tserver with Master: 94e58704a52c4792a750aa74ba470b7e (127.3.246.66:44143)
I20260924 07:19:13.827538 13306 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.246.66:40831
W20260924 07:19:13.905158 13628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:13.905376 13628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:13.905411 13628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:13.906759 13628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:13.906822 13628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:13.906863 13628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:19:13.908236 13628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:36045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:13.908483 13628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:13.908771 13628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:13.913594 13633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:13.913604 13636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:13.913605 13634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:13.914031 13628 server_base.cc:1034] running on GCE node
I20260924 07:19:13.914247 13628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:13.914484 13628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:13.915652 13628 hybrid_clock.cc:648] HybridClock initialized: now 1790234353915638 us; error 27 us; skew 500 ppm
I20260924 07:19:13.916922 13628 webserver.cc:460] Webserver started at http://127.3.246.67:41305/ using document root <none> and password file <none>
I20260924 07:19:13.917167 13628 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:13.917243 13628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:13.917378 13628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:13.918533 13628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "83ffe1463b7d4ad4b0dda35b9b95fa4d"
format_stamp: "Formatted at 2026-09-24 07:19:13 on dist-test-slave-rc1w"
server_key: "4a070aac7ce5823bf10096ed92a8386a"
server_key_iv: "bd70432a024c262689d529f5616563ca"
server_key_version: "encryptionkey@0"
I20260924 07:19:13.918908 13628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "83ffe1463b7d4ad4b0dda35b9b95fa4d"
format_stamp: "Formatted at 2026-09-24 07:19:13 on dist-test-slave-rc1w"
server_key: "4a070aac7ce5823bf10096ed92a8386a"
server_key_iv: "bd70432a024c262689d529f5616563ca"
server_key_version: "encryptionkey@0"
I20260924 07:19:13.920241 13628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:19:13.921015 13642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:13.921216 13628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:19:13.921312 13628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "83ffe1463b7d4ad4b0dda35b9b95fa4d"
format_stamp: "Formatted at 2026-09-24 07:19:13 on dist-test-slave-rc1w"
server_key: "4a070aac7ce5823bf10096ed92a8386a"
server_key_iv: "bd70432a024c262689d529f5616563ca"
server_key_version: "encryptionkey@0"
I20260924 07:19:13.921403 13628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:13.935015 13628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:13.935350 13628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:13.935503 13628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:13.935731 13628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:13.936048 13628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:13.936100 13628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:13.936141 13628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:13.936169 13628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:13.942092 13628 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:44587
I20260924 07:19:13.942133 13755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:44587 every 8 connection(s)
I20260924 07:19:13.942467 13628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:19:13.946364 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 13628
I20260924 07:19:13.946465 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:19:13.946678 4057 external_mini_cluster.cc:1428] Setting key 602d208656cfa811db2abcc7b8821240
I20260924 07:19:13.947932 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:13.943256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:56471 (local address 127.3.246.126:36045)
0924 07:19:13.943318 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:13.943320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:13.943521 (+ 201us) server_negotiation.cc:407] Connection header received
0924 07:19:13.944078 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:13.944080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:13.944094 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:13.944152 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:13.944830 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.945337 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.946463 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.946758 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.946849 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:13.947704 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:13.947706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:13.947706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:19:13.947711 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:13.947728 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:19:13.947732 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:13.947798 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:13.947852 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 07:19:13.947876 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 07:19:13.948179 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:13.943229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:36045 (local address 127.3.246.67:56471)
0924 07:19:13.943398 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:13.943432 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:13.943965 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:13.944220 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:13.944224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:13.944235 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:13.944723 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:13.944727 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.945431 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.945432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:13.946336 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:13.946339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:13.946854 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:13.946856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:13.947005 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:13.947539 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:19:13.947540 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:13.947575 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:19:13.947613 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:13.947801 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:13.947804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:13.947806 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:19:13.947853 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 07:19:13.947931 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260924 07:19:13.948726 13756 heartbeater.cc:347] Connected to a master server at 127.3.246.126:36045
I20260924 07:19:13.948828 13756 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:13.949013 13756 heartbeater.cc:511] Master 127.3.246.126:36045 requested a full tablet report, sending...
I20260924 07:19:13.949363 13306 ts_manager.cc:194] Registered new tserver with Master: 83ffe1463b7d4ad4b0dda35b9b95fa4d (127.3.246.67:44587)
I20260924 07:19:13.949790 13306 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.246.67:56471
I20260924 07:19:13.958431 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:19:13.960259 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 13366
W20260924 07:19:13.960273 13365 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:13.959720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.3.246.126:36045)
0924 07:19:13.959802 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:13.959805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:13.959905 (+ 100us) server_negotiation.cc:407] Connection header received
0924 07:19:13.959944 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:13.959945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:13.959956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:13.960007 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:13.960181 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37662: BlockingRecv error: recv got EOF from 127.0.0.1:37662 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:19:13.964601 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 13497
I20260924 07:19:13.968737 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 13628
I20260924 07:19:13.972965 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 13275
2026-09-24T07:19:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (584 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T07:19:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:19:13Z Disabled control of system clock
I20260924 07:19:13.999583 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:44765
--webserver_interface=127.3.246.126
--webserver_port=0
--builtin_ntp_servers=127.3.246.84:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:44765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:19:14.079120 13767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:14.079319 13767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:14.079339 13767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:14.080394 13767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:19:14.080428 13767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:14.080440 13767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:14.080453 13767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:19:14.080466 13767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:19:14.081727 13767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:44765
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:44765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:14.081912 13767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:14.082131 13767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:14.086854 13773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:14.086840 13775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:14.086829 13772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:14.087286 13767 server_base.cc:1034] running on GCE node
I20260924 07:19:14.087533 13767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:14.087821 13767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:14.088984 13767 hybrid_clock.cc:648] HybridClock initialized: now 1790234354088970 us; error 27 us; skew 500 ppm
I20260924 07:19:14.090286 13767 webserver.cc:460] Webserver started at http://127.3.246.126:35357/ using document root <none> and password file <none>
I20260924 07:19:14.090539 13767 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:14.090613 13767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:14.090801 13767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:14.091948 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "e497a14eb9004a97b1748e3d42709f29"
format_stamp: "Formatted at 2026-09-24 07:19:14 on dist-test-slave-rc1w"
server_key: "3213ceb2e1fc9ced5333e9720aa759a5"
server_key_iv: "14fbbfc46de3eae4c906e3bba6878ae1"
server_key_version: "encryptionkey@0"
I20260924 07:19:14.092329 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "e497a14eb9004a97b1748e3d42709f29"
format_stamp: "Formatted at 2026-09-24 07:19:14 on dist-test-slave-rc1w"
server_key: "3213ceb2e1fc9ced5333e9720aa759a5"
server_key_iv: "14fbbfc46de3eae4c906e3bba6878ae1"
server_key_version: "encryptionkey@0"
I20260924 07:19:14.093744 13767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:19:14.094485 13781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:14.094650 13767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:19:14.094767 13767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "e497a14eb9004a97b1748e3d42709f29"
format_stamp: "Formatted at 2026-09-24 07:19:14 on dist-test-slave-rc1w"
server_key: "3213ceb2e1fc9ced5333e9720aa759a5"
server_key_iv: "14fbbfc46de3eae4c906e3bba6878ae1"
server_key_version: "encryptionkey@0"
I20260924 07:19:14.094846 13767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:14.098479 13767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:14.098766 13767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:14.098918 13767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:14.103093 13767 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:44765
I20260924 07:19:14.103139 13833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:44765 every 8 connection(s)
I20260924 07:19:14.103506 13767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:19:14.104139 13834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:14.106891 13834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29: Bootstrap starting.
I20260924 07:19:14.107462 13834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:14.107757 13834 log.cc:824] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:14.108486 13834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29: No bootstrap required, opened a new log
I20260924 07:19:14.110150 13834 raft_consensus.cc:348] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e497a14eb9004a97b1748e3d42709f29" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44765 } }
I20260924 07:19:14.110284 13834 raft_consensus.cc:374] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:14.110306 13834 raft_consensus.cc:729] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e497a14eb9004a97b1748e3d42709f29, State: Initialized, Role: FOLLOWER
I20260924 07:19:14.110451 13834 consensus_queue.cc:260] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e497a14eb9004a97b1748e3d42709f29" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44765 } }
I20260924 07:19:14.110524 13834 raft_consensus.cc:388] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:19:14.110546 13834 raft_consensus.cc:482] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:19:14.110574 13834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:14.111128 13834 raft_consensus.cc:504] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e497a14eb9004a97b1748e3d42709f29" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44765 } }
I20260924 07:19:14.111217 13834 leader_election.cc:302] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e497a14eb9004a97b1748e3d42709f29; no voters:
I20260924 07:19:14.111378 13834 leader_election.cc:288] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:19:14.111456 13837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:19:14.111640 13837 raft_consensus.cc:686] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [term 1 LEADER]: Becoming Leader. State: Replica: e497a14eb9004a97b1748e3d42709f29, State: Running, Role: LEADER
I20260924 07:19:14.111649 13834 sys_catalog.cc:564] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:19:14.111761 13837 consensus_queue.cc:237] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e497a14eb9004a97b1748e3d42709f29" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44765 } }
I20260924 07:19:14.112438 13838 sys_catalog.cc:455] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e497a14eb9004a97b1748e3d42709f29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e497a14eb9004a97b1748e3d42709f29" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44765 } } }
I20260924 07:19:14.112542 13838 sys_catalog.cc:458] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:14.112677 13839 sys_catalog.cc:455] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e497a14eb9004a97b1748e3d42709f29. Latest consensus state: current_term: 1 leader_uuid: "e497a14eb9004a97b1748e3d42709f29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e497a14eb9004a97b1748e3d42709f29" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44765 } } }
I20260924 07:19:14.112735 13839 sys_catalog.cc:458] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29 [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:14.112879 13846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:19:14.113332 13846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:19:14.113687 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 13767
I20260924 07:19:14.113796 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:19:14.113991 4057 external_mini_cluster.cc:1428] Setting key 1839e498cbd6b6c77919c358208d738f
I20260924 07:19:14.115173 13846 catalog_manager.cc:1347] Generated new cluster ID: 126b31963ed247ca990c3d95ccbcfdb2
I20260924 07:19:14.115237 13846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:19:14.123535 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:14.114439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.3.246.126:44765)
0924 07:19:14.114645 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:14.114654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:14.121975 (+ 7321us) server_negotiation.cc:407] Connection header received
0924 07:19:14.122113 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:14.122121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:14.122189 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:14.122344 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:14.123060 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:14.123064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:14.123064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:19:14.123068 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:14.123109 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:19:14.123122 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:14.123186 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:14.123275 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 07:19:14.123333 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260924 07:19:14.129640 13846 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:19:14.129851 13846 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:19:14.134975 13846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e497a14eb9004a97b1748e3d42709f29: Generated new TSK 0
I20260924 07:19:14.135161 13846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:19:14.136564 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:44765
--builtin_ntp_servers=127.3.246.84:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:19:14.218402 13858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:14.218602 13858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:14.218626 13858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:14.219792 13858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:14.219830 13858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:14.219856 13858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:19:14.221179 13858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:44765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:14.221427 13858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:14.221761 13858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:14.227272 13863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:14.227356 13864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:14.227447 13866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:14.227640 13858 server_base.cc:1034] running on GCE node
I20260924 07:19:14.227834 13858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:14.228072 13858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:14.229245 13858 hybrid_clock.cc:648] HybridClock initialized: now 1790234354229232 us; error 28 us; skew 500 ppm
I20260924 07:19:14.230638 13858 webserver.cc:460] Webserver started at http://127.3.246.65:40283/ using document root <none> and password file <none>
I20260924 07:19:14.230894 13858 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:14.230952 13858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:14.231105 13858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:14.232496 13858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "01c816131a88495bb076bc49721882d4"
format_stamp: "Formatted at 2026-09-24 07:19:14 on dist-test-slave-rc1w"
server_key: "f0a940b954e7d52d9b828ea3286204ae"
server_key_iv: "ad1c3116b819fe72e1c75dd5bdc5c14b"
server_key_version: "encryptionkey@0"
I20260924 07:19:14.232903 13858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "01c816131a88495bb076bc49721882d4"
format_stamp: "Formatted at 2026-09-24 07:19:14 on dist-test-slave-rc1w"
server_key: "f0a940b954e7d52d9b828ea3286204ae"
server_key_iv: "ad1c3116b819fe72e1c75dd5bdc5c14b"
server_key_version: "encryptionkey@0"
I20260924 07:19:14.234558 13858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260924 07:19:14.235457 13872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:14.235694 13858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:19:14.235785 13858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "01c816131a88495bb076bc49721882d4"
format_stamp: "Formatted at 2026-09-24 07:19:14 on dist-test-slave-rc1w"
server_key: "f0a940b954e7d52d9b828ea3286204ae"
server_key_iv: "ad1c3116b819fe72e1c75dd5bdc5c14b"
server_key_version: "encryptionkey@0"
I20260924 07:19:14.235858 13858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:14.240497 13858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:14.240769 13858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:14.240891 13858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:14.241082 13858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:14.241408 13858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:14.241447 13858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:14.241473 13858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:14.241492 13858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:14.248260 13858 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:35463
I20260924 07:19:14.248298 13985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:35463 every 8 connection(s)
I20260924 07:19:14.248659 13858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:19:14.251514 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 13858
I20260924 07:19:14.251613 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:19:14.251837 4057 external_mini_cluster.cc:1428] Setting key da836a937ecdff07b1a8a48902482e84
I20260924 07:19:14.251972 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:14.249415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:53655 (local address 127.3.246.126:44765)
0924 07:19:14.249470 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:14.249472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:14.249731 (+ 259us) server_negotiation.cc:407] Connection header received
0924 07:19:14.250284 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:14.250285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:14.250292 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:14.250364 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:14.251638 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:14.251640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:14.251640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:19:14.251646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:14.251665 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:19:14.251670 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:14.251751 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:14.251855 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 07:19:14.251870 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 07:19:14.252164 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:14.249409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:44765 (local address 127.3.246.65:53655)
0924 07:19:14.249596 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:14.249629 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:14.250175 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:14.250436 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:14.250440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:14.250449 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:14.251396 (+ 947us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:19:14.251397 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:14.251445 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:19:14.251483 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:14.251751 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:14.251758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:14.251760 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:19:14.251858 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 07:19:14.251925 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260924 07:19:14.252682 13986 heartbeater.cc:347] Connected to a master server at 127.3.246.126:44765
I20260924 07:19:14.252761 13986 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:14.252902 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:44765
--builtin_ntp_servers=127.3.246.84:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 07:19:14.252931 13986 heartbeater.cc:511] Master 127.3.246.126:44765 requested a full tablet report, sending...
I20260924 07:19:14.253417 13797 ts_manager.cc:194] Registered new tserver with Master: 01c816131a88495bb076bc49721882d4 (127.3.246.65:35463)
W20260924 07:19:14.333104 13989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:14.333348 13989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:14.333386 13989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:14.334575 13989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:14.334641 13989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:14.334682 13989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:19:14.336030 13989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:44765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:14.336290 13989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:14.336580 13989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:14.341455 13994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:14.341445 13997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:14.341611 13989 server_base.cc:1034] running on GCE node
W20260924 07:19:14.341434 13995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:14.341924 13989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:14.342139 13989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:14.343274 13989 hybrid_clock.cc:648] HybridClock initialized: now 1790234354343264 us; error 27 us; skew 500 ppm
I20260924 07:19:14.344549 13989 webserver.cc:460] Webserver started at http://127.3.246.66:45145/ using document root <none> and password file <none>
I20260924 07:19:14.344790 13989 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:14.344841 13989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:14.345012 13989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:14.346123 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "44ed4b72775c4c2696b15a866a560a16"
format_stamp: "Formatted at 2026-09-24 07:19:14 on dist-test-slave-rc1w"
server_key: "ec507b53397c2c8a0748000c63b5e54d"
server_key_iv: "9aae1f049a42e2499095631aeef4e780"
server_key_version: "encryptionkey@0"
I20260924 07:19:14.346484 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "44ed4b72775c4c2696b15a866a560a16"
format_stamp: "Formatted at 2026-09-24 07:19:14 on dist-test-slave-rc1w"
server_key: "ec507b53397c2c8a0748000c63b5e54d"
server_key_iv: "9aae1f049a42e2499095631aeef4e780"
server_key_version: "encryptionkey@0"
I20260924 07:19:14.347812 13989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:19:14.348549 14003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:14.348719 13989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:19:14.348816 13989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "44ed4b72775c4c2696b15a866a560a16"
format_stamp: "Formatted at 2026-09-24 07:19:14 on dist-test-slave-rc1w"
server_key: "ec507b53397c2c8a0748000c63b5e54d"
server_key_iv: "9aae1f049a42e2499095631aeef4e780"
server_key_version: "encryptionkey@0"
I20260924 07:19:14.348897 13989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:14.352200 13989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:14.352450 13989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:14.352591 13989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:14.352795 13989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:14.353091 13989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:14.353149 13989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:14.353188 13989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:14.353216 13989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:14.358824 13989 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:38481
I20260924 07:19:14.358902 14116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:38481 every 8 connection(s)
I20260924 07:19:14.359202 13989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:19:14.362143 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:14.359900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:41609 (local address 127.3.246.126:44765)
0924 07:19:14.359946 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:14.359949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:14.360141 (+ 192us) server_negotiation.cc:407] Connection header received
0924 07:19:14.360640 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:14.360642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:14.360649 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:14.360695 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:14.361906 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:14.361908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:14.361908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:19:14.361913 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:14.361929 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:19:14.361934 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:14.361987 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:14.362059 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 07:19:14.362074 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:19:14.362334 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:14.359867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:44765 (local address 127.3.246.66:41609)
0924 07:19:14.360028 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:14.360059 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:14.360532 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:14.360756 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:14.360759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:14.360769 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:14.361728 (+ 959us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:19:14.361729 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:14.361766 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:19:14.361811 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:14.361983 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:14.361989 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:14.361992 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:19:14.362059 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:19:14.362121 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260924 07:19:14.362758 14117 heartbeater.cc:347] Connected to a master server at 127.3.246.126:44765
I20260924 07:19:14.362843 14117 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:14.362998 14117 heartbeater.cc:511] Master 127.3.246.126:44765 requested a full tablet report, sending...
I20260924 07:19:14.363338 13797 ts_manager.cc:194] Registered new tserver with Master: 44ed4b72775c4c2696b15a866a560a16 (127.3.246.66:38481)
I20260924 07:19:14.367532 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 13989
I20260924 07:19:14.367617 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:19:14.367756 4057 external_mini_cluster.cc:1428] Setting key c67a5179135606a02d622a26499fcf67
I20260924 07:19:14.368672 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:44765
--builtin_ntp_servers=127.3.246.84:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:19:14.448309 14120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:14.448518 14120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:14.448541 14120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:14.449652 14120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:14.449707 14120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:14.449735 14120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:19:14.450985 14120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:44765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:14.451231 14120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:14.451546 14120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:14.457019 14128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:14.457108 14125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:14.457036 14126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:14.457247 14120 server_base.cc:1034] running on GCE node
I20260924 07:19:14.457496 14120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:14.457799 14120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:14.458956 14120 hybrid_clock.cc:648] HybridClock initialized: now 1790234354458941 us; error 35 us; skew 500 ppm
I20260924 07:19:14.460181 14120 webserver.cc:460] Webserver started at http://127.3.246.67:46539/ using document root <none> and password file <none>
I20260924 07:19:14.460443 14120 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:14.460511 14120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:14.460669 14120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:14.461830 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "008b84dca4114e17a8718a7b0dcbbf8c"
format_stamp: "Formatted at 2026-09-24 07:19:14 on dist-test-slave-rc1w"
server_key: "97c8b581559112628de59d9aeac049a1"
server_key_iv: "f9fb8aaaf83ecb6a9c311d96956b9b90"
server_key_version: "encryptionkey@0"
I20260924 07:19:14.462181 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "008b84dca4114e17a8718a7b0dcbbf8c"
format_stamp: "Formatted at 2026-09-24 07:19:14 on dist-test-slave-rc1w"
server_key: "97c8b581559112628de59d9aeac049a1"
server_key_iv: "f9fb8aaaf83ecb6a9c311d96956b9b90"
server_key_version: "encryptionkey@0"
I20260924 07:19:14.463424 14120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:19:14.464126 14134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:14.464300 14120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:19:14.464397 14120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "008b84dca4114e17a8718a7b0dcbbf8c"
format_stamp: "Formatted at 2026-09-24 07:19:14 on dist-test-slave-rc1w"
server_key: "97c8b581559112628de59d9aeac049a1"
server_key_iv: "f9fb8aaaf83ecb6a9c311d96956b9b90"
server_key_version: "encryptionkey@0"
I20260924 07:19:14.464478 14120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:14.467814 14120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:14.468057 14120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:14.468196 14120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:14.468402 14120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:14.468696 14120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:14.468751 14120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:14.468791 14120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:14.468818 14120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:14.474699 14120 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:38449
I20260924 07:19:14.474731 14247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:38449 every 8 connection(s)
I20260924 07:19:14.475049 14120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:19:14.477962 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:14.475789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:41617 (local address 127.3.246.126:44765)
0924 07:19:14.475836 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:14.475839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:14.476068 (+ 229us) server_negotiation.cc:407] Connection header received
0924 07:19:14.476506 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:14.476507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:14.476512 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:14.476566 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:14.477752 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:14.477753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:14.477754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:19:14.477757 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:14.477771 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:19:14.477775 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:14.477830 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:14.477899 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 07:19:14.477911 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:19:14.478148 14250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:14.475786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:44765 (local address 127.3.246.67:41617)
0924 07:19:14.475956 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:14.475987 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:14.476418 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:14.476641 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:14.476644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:14.476652 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:14.477532 (+ 880us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:19:14.477532 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:14.477559 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:19:14.477585 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:14.477841 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:14.477845 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:14.477848 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:19:14.477908 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 07:19:14.477955 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260924 07:19:14.478619 14248 heartbeater.cc:347] Connected to a master server at 127.3.246.126:44765
I20260924 07:19:14.478691 14248 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:14.478856 14248 heartbeater.cc:511] Master 127.3.246.126:44765 requested a full tablet report, sending...
I20260924 07:19:14.479210 13797 ts_manager.cc:194] Registered new tserver with Master: 008b84dca4114e17a8718a7b0dcbbf8c (127.3.246.67:38449)
I20260924 07:19:14.483119 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 14120
I20260924 07:19:14.483191 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:19:14.483331 4057 external_mini_cluster.cc:1428] Setting key bde29fab7fbb3848a7cfb7b0c0ea638b
I20260924 07:19:14.484180 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:19:14.485846 13856 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:14.485394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.3.246.126:44765)
0924 07:19:14.485461 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:14.485463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:14.485530 (+ 67us) server_negotiation.cc:407] Connection header received
0924 07:19:14.485566 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:14.485567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:14.485570 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:14.485615 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:14.485790 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60142: BlockingRecv error: recv got EOF from 127.0.0.1:60142 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:19:14.485909 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 13858
I20260924 07:19:14.490875 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 13989
I20260924 07:19:14.495190 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 14120
I20260924 07:19:14.506179 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 13767
2026-09-24T07:19:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (533 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:19:14 dist-test-slave-rc1w krb5kdc[14258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:19:14 dist-test-slave-rc1w krb5kdc[14258](info): set up 2 sockets
Sep 24 07:19:14 dist-test-slave-rc1w krb5kdc[14258](info): commencing operation
krb5kdc: starting...
W20260924 07:19:15.547109 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:19:15 dist-test-slave-rc1w krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:19:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:19:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:19:15.664940 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:46725
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:46725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:19:15.745023 14271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:15.745224 14271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:15.745246 14271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:15.746461 14271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:19:15.746503 14271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:15.746517 14271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:15.746531 14271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:19:15.746542 14271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:19:15.747994 14271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:46725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:46725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:15.748241 14271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:15.748476 14271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:15.753324 14279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:15.753497 14271 server_base.cc:1034] running on GCE node
W20260924 07:19:15.753338 14276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:15.753355 14277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:15.753886 14271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:15.754176 14271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:15.755344 14271 hybrid_clock.cc:648] HybridClock initialized: now 1790234355755328 us; error 27 us; skew 500 ppm
Sep 24 07:19:15 dist-test-slave-rc1w krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:15.757937 14271 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:19:15.758399 14271 webserver.cc:460] Webserver started at http://127.3.246.126:38513/ using document root <none> and password file <none>
I20260924 07:19:15.758620 14271 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:15.758687 14271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:15.758838 14271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:15.759871 14271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "13577bcafc814e588e981f36c746dfb0"
format_stamp: "Formatted at 2026-09-24 07:19:15 on dist-test-slave-rc1w"
server_key: "dbead3bea5ce8f093a25077fd85b8778"
server_key_iv: "7785694e49b9d1b2dd6396c0dabe460c"
server_key_version: "encryptionkey@0"
I20260924 07:19:15.760229 14271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "13577bcafc814e588e981f36c746dfb0"
format_stamp: "Formatted at 2026-09-24 07:19:15 on dist-test-slave-rc1w"
server_key: "dbead3bea5ce8f093a25077fd85b8778"
server_key_iv: "7785694e49b9d1b2dd6396c0dabe460c"
server_key_version: "encryptionkey@0"
I20260924 07:19:15.761554 14271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:19:15.762378 14286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:15.762598 14271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:19:15.762712 14271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "13577bcafc814e588e981f36c746dfb0"
format_stamp: "Formatted at 2026-09-24 07:19:15 on dist-test-slave-rc1w"
server_key: "dbead3bea5ce8f093a25077fd85b8778"
server_key_iv: "7785694e49b9d1b2dd6396c0dabe460c"
server_key_version: "encryptionkey@0"
I20260924 07:19:15.762802 14271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:15.788604 14271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:15.790722 14271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:15.790902 14271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:15.795121 14271 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:46725
I20260924 07:19:15.795159 14338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:46725 every 8 connection(s)
I20260924 07:19:15.795540 14271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:19:15.796528 14339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:15.798801 14339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0: Bootstrap starting.
I20260924 07:19:15.799324 14339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:15.799575 14339 log.cc:824] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:15.799758 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 14271
I20260924 07:19:15.799875 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:19:15.800043 4057 external_mini_cluster.cc:1428] Setting key f1c0f9948fe4a523100f2d55f271ad52
I20260924 07:19:15.800343 14339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0: No bootstrap required, opened a new log
I20260924 07:19:15.802224 14339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13577bcafc814e588e981f36c746dfb0" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46725 } }
I20260924 07:19:15.802359 14339 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:15.802382 14339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13577bcafc814e588e981f36c746dfb0, State: Initialized, Role: FOLLOWER
I20260924 07:19:15.802574 14339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13577bcafc814e588e981f36c746dfb0" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46725 } }
I20260924 07:19:15.802666 14339 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:19:15.802703 14339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:19:15.802754 14339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:15.803745 14339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13577bcafc814e588e981f36c746dfb0" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46725 } }
I20260924 07:19:15.803840 14339 leader_election.cc:302] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13577bcafc814e588e981f36c746dfb0; no voters:
I20260924 07:19:15.804018 14339 leader_election.cc:288] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:19:15.804069 14344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:19:15.804216 14344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [term 1 LEADER]: Becoming Leader. State: Replica: 13577bcafc814e588e981f36c746dfb0, State: Running, Role: LEADER
I20260924 07:19:15.804268 14339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:19:15.804345 14344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13577bcafc814e588e981f36c746dfb0" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46725 } }
I20260924 07:19:15.804788 14345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13577bcafc814e588e981f36c746dfb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13577bcafc814e588e981f36c746dfb0" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46725 } } }
I20260924 07:19:15.804864 14345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:15.805115 14346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13577bcafc814e588e981f36c746dfb0. Latest consensus state: current_term: 1 leader_uuid: "13577bcafc814e588e981f36c746dfb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13577bcafc814e588e981f36c746dfb0" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46725 } } }
I20260924 07:19:15.805218 14346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:15.805450 14352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:19:15.805996 14352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:19:15.807520 14352 catalog_manager.cc:1347] Generated new cluster ID: 710b3bae4e9241dda54cc5a9a50319d0
I20260924 07:19:15.807571 14352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:19:15 dist-test-slave-rc1w krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:19:15.812866 14352 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:19:15.812876 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:15.800533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.3.246.126:46725)
0924 07:19:15.800709 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:15.800720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:15.800751 (+ 31us) server_negotiation.cc:407] Connection header received
0924 07:19:15.802220 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:15.802228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:15.802281 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:15.802453 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:15.803216 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:15.804375 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:15.805480 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:15.806242 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:15.806350 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:15.809146 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:15.809150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:15.809150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:15.809161 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:15.811136 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:15.811549 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:15.811552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:15.811552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:15.811682 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:15.811940 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:15.811941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:15.811942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:15.812219 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:15.812370 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:15.812519 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 07:19:15.812672 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260924 07:19:15.813556 14352 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:19:15.818265 14352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0: Generated new TSK 0
I20260924 07:19:15.818437 14352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:19:15.846629 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46725
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:19:15.927544 14365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:15.927778 14365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:15.927817 14365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:15.928956 14365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:15.929018 14365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:15.929064 14365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:19:15.930577 14365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:15.930835 14365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:15.931058 14365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:15.936648 14370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:15.936650 14373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:15.936940 14371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:15.937107 14365 server_base.cc:1034] running on GCE node
I20260924 07:19:15.937310 14365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:15.937505 14365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:15.938670 14365 hybrid_clock.cc:648] HybridClock initialized: now 1790234355938659 us; error 41 us; skew 500 ppm
Sep 24 07:19:15 dist-test-slave-rc1w krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:15.941077 14365 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:19:15.941540 14365 webserver.cc:460] Webserver started at http://127.3.246.65:42577/ using document root <none> and password file <none>
I20260924 07:19:15.941823 14365 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:15.941890 14365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:15.942046 14365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:15.943253 14365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "3c194d23ab4040859716cc25356ee71e"
format_stamp: "Formatted at 2026-09-24 07:19:15 on dist-test-slave-rc1w"
server_key: "c254bac57d308213b04fb23345e2a910"
server_key_iv: "7ba0ab17fe8ad8617fa0359508b123d5"
server_key_version: "encryptionkey@0"
I20260924 07:19:15.943611 14365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "3c194d23ab4040859716cc25356ee71e"
format_stamp: "Formatted at 2026-09-24 07:19:15 on dist-test-slave-rc1w"
server_key: "c254bac57d308213b04fb23345e2a910"
server_key_iv: "7ba0ab17fe8ad8617fa0359508b123d5"
server_key_version: "encryptionkey@0"
I20260924 07:19:15.944921 14365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:19:15.945595 14380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:15.945784 14365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:19:15.945890 14365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "3c194d23ab4040859716cc25356ee71e"
format_stamp: "Formatted at 2026-09-24 07:19:15 on dist-test-slave-rc1w"
server_key: "c254bac57d308213b04fb23345e2a910"
server_key_iv: "7ba0ab17fe8ad8617fa0359508b123d5"
server_key_version: "encryptionkey@0"
I20260924 07:19:15.945974 14365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:15.956637 14365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:15.958964 14365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:15.959125 14365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:15.959353 14365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:15.959661 14365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:15.959721 14365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:15.959762 14365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:15.959790 14365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:15.965817 14365 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:45731
I20260924 07:19:15.965878 14493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:45731 every 8 connection(s)
I20260924 07:19:15.966197 14365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:19:15 dist-test-slave-rc1w krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:19:15.972074 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 14365
I20260924 07:19:15.972152 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:19:15.972362 4057 external_mini_cluster.cc:1428] Setting key e87e90ef571aa8399a6598196fc8833a
I20260924 07:19:15.975791 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:15.966952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:51631 (local address 127.3.246.126:46725)
0924 07:19:15.967024 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:15.967027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:15.967238 (+ 211us) server_negotiation.cc:407] Connection header received
0924 07:19:15.967692 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:15.967693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:15.967703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:15.967754 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:15.968612 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:15.969108 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:15.970060 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:15.970262 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:15.970351 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:15.972450 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:15.972453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:15.972453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:15.972461 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:15.974309 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:15.974810 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:15.974811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:15.974811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:15.974869 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:15.975149 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:15.975150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:15.975150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:15.975251 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:15.975314 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:15.975658 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 07:19:15.975728 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:19:15.976048 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:15.966943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46725 (local address 127.3.246.65:51631)
0924 07:19:15.967113 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:15.967152 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:15.967616 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:15.967806 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:15.967809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:15.968358 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:15.968540 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:15.968544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:15.969203 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:15.969203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:15.969994 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:15.969996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:15.970361 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:15.970361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:15.970456 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:15.970881 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:15.970888 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:15.972351 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:15.974398 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:15.974404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:15.974406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:15.974716 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:15.974940 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:15.974940 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:15.974941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:15.975085 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:15.975325 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:15.975330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:15.975492 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 07:19:15.975620 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 07:19:15.975795 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260924 07:19:15.976612 14494 heartbeater.cc:347] Connected to a master server at 127.3.246.126:46725
I20260924 07:19:15.976707 14494 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:15.976897 14494 heartbeater.cc:511] Master 127.3.246.126:46725 requested a full tablet report, sending...
I20260924 07:19:15.977392 14303 ts_manager.cc:194] Registered new tserver with Master: 3c194d23ab4040859716cc25356ee71e (127.3.246.65:45731)
I20260924 07:19:15.978044 14303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:51631
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:19:16.000181 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46725
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:19:16.079942 14499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:16.080183 14499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:16.080219 14499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:16.081420 14499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:16.081489 14499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:16.081532 14499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:19:16.083017 14499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:16.083289 14499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:16.083510 14499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:16.088444 14507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:16.088503 14499 server_base.cc:1034] running on GCE node
W20260924 07:19:16.088452 14505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:16.088454 14504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:16.088883 14499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:16.089140 14499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:16.090307 14499 hybrid_clock.cc:648] HybridClock initialized: now 1790234356090287 us; error 32 us; skew 500 ppm
Sep 24 07:19:16 dist-test-slave-rc1w krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:16.092774 14499 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:19:16.093247 14499 webserver.cc:460] Webserver started at http://127.3.246.66:33537/ using document root <none> and password file <none>
I20260924 07:19:16.093492 14499 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:16.093560 14499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:16.093739 14499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:16.094794 14499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "d9d01d73deda449d9fb0ac770c395bba"
format_stamp: "Formatted at 2026-09-24 07:19:16 on dist-test-slave-rc1w"
server_key: "143e6ccb450e481bb335e0251429b8d0"
server_key_iv: "cd8a06ea2426ce3f4b6b55d1d5fca6d4"
server_key_version: "encryptionkey@0"
I20260924 07:19:16.095171 14499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "d9d01d73deda449d9fb0ac770c395bba"
format_stamp: "Formatted at 2026-09-24 07:19:16 on dist-test-slave-rc1w"
server_key: "143e6ccb450e481bb335e0251429b8d0"
server_key_iv: "cd8a06ea2426ce3f4b6b55d1d5fca6d4"
server_key_version: "encryptionkey@0"
I20260924 07:19:16.096474 14499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:19:16.097195 14514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:16.097388 14499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:19:16.097494 14499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "d9d01d73deda449d9fb0ac770c395bba"
format_stamp: "Formatted at 2026-09-24 07:19:16 on dist-test-slave-rc1w"
server_key: "143e6ccb450e481bb335e0251429b8d0"
server_key_iv: "cd8a06ea2426ce3f4b6b55d1d5fca6d4"
server_key_version: "encryptionkey@0"
I20260924 07:19:16.097577 14499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:16.115566 14499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:16.117589 14499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:16.117815 14499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:16.118050 14499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:16.118383 14499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:16.118448 14499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:16.118491 14499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:16.118520 14499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:16.124336 14499 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:41957
I20260924 07:19:16.124388 14627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:41957 every 8 connection(s)
I20260924 07:19:16.124745 14499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:19:16 dist-test-slave-rc1w krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:19:16.134282 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.125485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:43403 (local address 127.3.246.126:46725)
0924 07:19:16.125541 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:16.125544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:16.125770 (+ 226us) server_negotiation.cc:407] Connection header received
0924 07:19:16.126269 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:16.126271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:16.126281 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:16.126333 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:16.127282 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.127811 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.128780 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.128979 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.129039 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.131272 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:16.131273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:16.131274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:16.131280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:16.132821 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:16.133227 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:16.133228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:16.133228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:16.133279 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:16.133488 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:16.133488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:16.133489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:16.133589 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:16.133663 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:16.134131 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 07:19:16.134217 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:19:16.134454 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.125466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46725 (local address 127.3.246.66:43403)
0924 07:19:16.125623 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:16.125658 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:16.126181 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:16.126385 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:16.126388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:16.126991 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:16.127186 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.127190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.127895 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.127896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.128696 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.128698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.129057 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.129058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.129172 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.129660 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:16.129733 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:16.131172 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:16.132931 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:16.132938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:16.132940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:16.133156 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:16.133337 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:16.133337 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:16.133338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:16.133432 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:16.133774 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:16.133781 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:16.133975 (+ 194us) client_negotiation.cc:769] Sending connection context
0924 07:19:16.134081 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 07:19:16.134213 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260924 07:19:16.135010 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 14499
I20260924 07:19:16.135052 14628 heartbeater.cc:347] Connected to a master server at 127.3.246.126:46725
I20260924 07:19:16.135128 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:19:16.135143 14628 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:16.135334 14628 heartbeater.cc:511] Master 127.3.246.126:46725 requested a full tablet report, sending...
I20260924 07:19:16.135345 4057 external_mini_cluster.cc:1428] Setting key 3e1446e16f246231991fca0f3e0392fa
I20260924 07:19:16.135702 14303 ts_manager.cc:194] Registered new tserver with Master: d9d01d73deda449d9fb0ac770c395bba (127.3.246.66:41957)
I20260924 07:19:16.136399 14303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:43403
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:19:16.161976 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46725
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:19:16.242475 14633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:16.242676 14633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:16.242699 14633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:16.243768 14633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:16.243803 14633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:16.243829 14633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:19:16.245190 14633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:16.245376 14633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:16.245544 14633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:16.250413 14641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:16.250452 14638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:16.250407 14639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:16.250694 14633 server_base.cc:1034] running on GCE node
I20260924 07:19:16.250916 14633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:16.251155 14633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:16.252319 14633 hybrid_clock.cc:648] HybridClock initialized: now 1790234356252304 us; error 24 us; skew 500 ppm
Sep 24 07:19:16 dist-test-slave-rc1w krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:16.254738 14633 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:19:16.255167 14633 webserver.cc:460] Webserver started at http://127.3.246.67:46773/ using document root <none> and password file <none>
I20260924 07:19:16.255367 14633 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:16.255409 14633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:16.255494 14633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:16.256448 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "cc0f97beae4d4ab0a0cc533188a7ee77"
format_stamp: "Formatted at 2026-09-24 07:19:16 on dist-test-slave-rc1w"
server_key: "2649f9bdda4fb4f7dbfc4c6674f1abde"
server_key_iv: "93673366595714ff072aafd646e7d861"
server_key_version: "encryptionkey@0"
I20260924 07:19:16.256737 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "cc0f97beae4d4ab0a0cc533188a7ee77"
format_stamp: "Formatted at 2026-09-24 07:19:16 on dist-test-slave-rc1w"
server_key: "2649f9bdda4fb4f7dbfc4c6674f1abde"
server_key_iv: "93673366595714ff072aafd646e7d861"
server_key_version: "encryptionkey@0"
I20260924 07:19:16.257982 14633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:19:16.258695 14648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:16.258870 14633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:19:16.258952 14633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "cc0f97beae4d4ab0a0cc533188a7ee77"
format_stamp: "Formatted at 2026-09-24 07:19:16 on dist-test-slave-rc1w"
server_key: "2649f9bdda4fb4f7dbfc4c6674f1abde"
server_key_iv: "93673366595714ff072aafd646e7d861"
server_key_version: "encryptionkey@0"
I20260924 07:19:16.259003 14633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:16.273077 14633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:16.275353 14633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:16.275461 14633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:16.275624 14633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:16.275928 14633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:16.275969 14633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:16.275988 14633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:16.276036 14633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:16.281646 14633 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:41695
I20260924 07:19:16.281766 14761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:41695 every 8 connection(s)
I20260924 07:19:16.282095 14633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:19:16.286595 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 14633
I20260924 07:19:16.286715 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:19:16.286906 4057 external_mini_cluster.cc:1428] Setting key 0c63d397f0659eddf1d6664c5edb81f4
Sep 24 07:19:16 dist-test-slave-rc1w krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:19:16.291503 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.282877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:48645 (local address 127.3.246.126:46725)
0924 07:19:16.282959 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:16.282962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:16.283142 (+ 180us) server_negotiation.cc:407] Connection header received
0924 07:19:16.283679 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:16.283681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:16.283693 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:16.283744 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:16.284663 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.285191 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.286264 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.286466 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.286551 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.288704 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:16.288705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:16.288705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:16.288714 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:16.290134 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:16.290555 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:16.290556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:16.290557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:16.290614 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:16.290866 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:16.290867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:16.290867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:16.290978 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:16.291055 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:16.291338 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 07:19:16.291413 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:19:16.291704 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.282842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46725 (local address 127.3.246.67:48645)
0924 07:19:16.283029 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:16.283064 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:16.283563 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:16.283813 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:16.283818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:16.284380 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:16.284564 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.284568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.285299 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.285300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.286174 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.286176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.286561 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.286562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.286738 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.287312 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:16.287319 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:16.288606 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:16.290239 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:16.290246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:16.290248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:16.290468 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:16.290680 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:16.290681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:16.290681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:16.290795 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:16.291047 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:16.291053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:16.291190 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 07:19:16.291312 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 07:19:16.291466 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260924 07:19:16.292136 14762 heartbeater.cc:347] Connected to a master server at 127.3.246.126:46725
I20260924 07:19:16.292212 14762 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:16.292344 14762 heartbeater.cc:511] Master 127.3.246.126:46725 requested a full tablet report, sending...
I20260924 07:19:16.292646 14303 ts_manager.cc:194] Registered new tserver with Master: cc0f97beae4d4ab0a0cc533188a7ee77 (127.3.246.67:41695)
I20260924 07:19:16.292975 14303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:48645
I20260924 07:19:16.298564 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:19:16.306407 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.299924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42346 (local address 127.3.246.126:46725)
0924 07:19:16.299983 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:16.299985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:16.300091 (+ 106us) server_negotiation.cc:407] Connection header received
0924 07:19:16.300160 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:16.300161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:16.300171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:16.300258 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:16.300844 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.301310 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.302131 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.302318 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.302373 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.303560 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:16.303562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:16.303562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:16.303567 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:16.304990 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:16.305472 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:16.305473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:16.305474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:16.305525 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:16.305803 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:16.305804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:16.305804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:16.305943 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:16.306019 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:16.306242 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 07:19:16.306309 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:19:16.308009 14303 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:19:16.309510 14303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:19:16.316401 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.311793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:41957 (local address 127.0.0.1:48840)
0924 07:19:16.311988 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:16.312028 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:16.312092 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:16.312577 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:16.312580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:16.312582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:16.312654 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.312655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.314059 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.314060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.315227 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.315230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.316076 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.316077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.316156 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.316170 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 07:19:16.316250 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:19:16.316313 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":149,"thread_start_us":87,"threads_started":1}
I20260924 07:19:16.316401 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.311917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.3.246.66:41957)
0924 07:19:16.312416 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:16.312420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:16.312430 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:19:16.312468 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:16.312470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:16.312488 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:16.312573 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:16.312772 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.313943 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.315350 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.315985 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.316062 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.316127 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:16.316277 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 07:19:16.316319 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":89,"threads_started":1}
I20260924 07:19:16.316655 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.311836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:41695 (local address 127.0.0.1:55042)
0924 07:19:16.312317 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:16.312328 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:16.312450 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:16.312777 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:16.312778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:16.312780 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:16.312827 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.312828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.314096 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.314097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.315581 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.315584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.316416 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.316417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.316479 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.316483 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:19:16.316539 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 07:19:16.316585 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":126,"thread_start_us":65,"threads_started":1}
I20260924 07:19:16.316804 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.312037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:45731 (local address 127.0.0.1:56078)
0924 07:19:16.312858 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:16.312870 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:16.312917 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:16.313132 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:16.313133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:16.313135 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:16.313170 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.313171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.314110 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.314111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.315582 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.315584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.316558 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.316559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.316633 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.316637 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:19:16.316686 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 07:19:16.316730 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":165,"thread_start_us":50,"threads_started":1}
I20260924 07:19:16.316900 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.312107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.3.246.67:41695)
0924 07:19:16.312295 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:16.312301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:16.312416 (+ 115us) server_negotiation.cc:407] Connection header received
0924 07:19:16.312593 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:16.312603 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:16.312619 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:16.312695 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:16.312913 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.313994 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.315685 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.316274 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.316332 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.316383 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:16.316763 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 07:19:16.316810 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":104,"threads_started":1}
I20260924 07:19:16.317085 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.312163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56078 (local address 127.3.246.65:45731)
0924 07:19:16.312370 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:16.312374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:16.312978 (+ 604us) server_negotiation.cc:407] Connection header received
0924 07:19:16.313013 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:16.313018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:16.313030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:16.313133 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:16.313227 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.313974 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.315669 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.316395 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.316465 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.316524 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:16.316956 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 07:19:16.317002 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":105,"threads_started":1}
I20260924 07:19:16.317332 14562 tablet_service.cc:1467] Processing CreateTablet for tablet ff3f143efa774b2dabfceaf096bbe3b2 (DEFAULT_TABLE table=test-table [id=b3dcffdb56e24accb86d843d3a586d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:19:16.317494 14696 tablet_service.cc:1467] Processing CreateTablet for tablet ff3f143efa774b2dabfceaf096bbe3b2 (DEFAULT_TABLE table=test-table [id=b3dcffdb56e24accb86d843d3a586d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:19:16.317835 14562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3f143efa774b2dabfceaf096bbe3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:16.317901 14696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3f143efa774b2dabfceaf096bbe3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:16.318082 14428 tablet_service.cc:1467] Processing CreateTablet for tablet ff3f143efa774b2dabfceaf096bbe3b2 (DEFAULT_TABLE table=test-table [id=b3dcffdb56e24accb86d843d3a586d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:19:16.318324 14428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3f143efa774b2dabfceaf096bbe3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:16.320966 14776 tablet_bootstrap.cc:492] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e: Bootstrap starting.
I20260924 07:19:16.321424 14778 tablet_bootstrap.cc:492] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77: Bootstrap starting.
I20260924 07:19:16.321583 14777 tablet_bootstrap.cc:492] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba: Bootstrap starting.
I20260924 07:19:16.321732 14776 tablet_bootstrap.cc:654] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:16.322184 14778 tablet_bootstrap.cc:654] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:16.322212 14777 tablet_bootstrap.cc:654] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:16.322225 14776 log.cc:824] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:16.322832 14777 log.cc:824] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:16.322832 14778 log.cc:824] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:16.323006 14776 tablet_bootstrap.cc:492] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e: No bootstrap required, opened a new log
I20260924 07:19:16.323118 14776 ts_tablet_manager.cc:1397] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:19:16.323734 14778 tablet_bootstrap.cc:492] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77: No bootstrap required, opened a new log
I20260924 07:19:16.323753 14777 tablet_bootstrap.cc:492] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba: No bootstrap required, opened a new log
I20260924 07:19:16.323817 14778 ts_tablet_manager.cc:1397] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:19:16.323817 14777 ts_tablet_manager.cc:1397] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:19:16.324712 14776 raft_consensus.cc:348] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d01d73deda449d9fb0ac770c395bba" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41957 } } peers { permanent_uuid: "3c194d23ab4040859716cc25356ee71e" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45731 } } peers { permanent_uuid: "cc0f97beae4d4ab0a0cc533188a7ee77" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41695 } }
I20260924 07:19:16.324872 14776 raft_consensus.cc:374] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:16.324934 14776 raft_consensus.cc:729] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c194d23ab4040859716cc25356ee71e, State: Initialized, Role: FOLLOWER
I20260924 07:19:16.325066 14776 consensus_queue.cc:260] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d01d73deda449d9fb0ac770c395bba" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41957 } } peers { permanent_uuid: "3c194d23ab4040859716cc25356ee71e" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45731 } } peers { permanent_uuid: "cc0f97beae4d4ab0a0cc533188a7ee77" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41695 } }
I20260924 07:19:16.325361 14776 ts_tablet_manager.cc:1428] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:19:16.325496 14494 heartbeater.cc:503] Master 127.3.246.126:46725 was elected leader, sending a full tablet report...
I20260924 07:19:16.325825 14777 raft_consensus.cc:348] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d01d73deda449d9fb0ac770c395bba" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41957 } } peers { permanent_uuid: "3c194d23ab4040859716cc25356ee71e" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45731 } } peers { permanent_uuid: "cc0f97beae4d4ab0a0cc533188a7ee77" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41695 } }
I20260924 07:19:16.325795 14778 raft_consensus.cc:348] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d01d73deda449d9fb0ac770c395bba" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41957 } } peers { permanent_uuid: "3c194d23ab4040859716cc25356ee71e" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45731 } } peers { permanent_uuid: "cc0f97beae4d4ab0a0cc533188a7ee77" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41695 } }
I20260924 07:19:16.325950 14778 raft_consensus.cc:374] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:16.325999 14778 raft_consensus.cc:729] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0f97beae4d4ab0a0cc533188a7ee77, State: Initialized, Role: FOLLOWER
I20260924 07:19:16.326030 14777 raft_consensus.cc:374] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:16.326094 14777 raft_consensus.cc:729] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d01d73deda449d9fb0ac770c395bba, State: Initialized, Role: FOLLOWER
I20260924 07:19:16.326140 14778 consensus_queue.cc:260] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d01d73deda449d9fb0ac770c395bba" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41957 } } peers { permanent_uuid: "3c194d23ab4040859716cc25356ee71e" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45731 } } peers { permanent_uuid: "cc0f97beae4d4ab0a0cc533188a7ee77" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41695 } }
I20260924 07:19:16.326247 14777 consensus_queue.cc:260] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d01d73deda449d9fb0ac770c395bba" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41957 } } peers { permanent_uuid: "3c194d23ab4040859716cc25356ee71e" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45731 } } peers { permanent_uuid: "cc0f97beae4d4ab0a0cc533188a7ee77" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41695 } }
I20260924 07:19:16.326419 14778 ts_tablet_manager.cc:1428] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 07:19:16.326563 14777 ts_tablet_manager.cc:1428] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:19:16.326583 14762 heartbeater.cc:503] Master 127.3.246.126:46725 was elected leader, sending a full tablet report...
I20260924 07:19:16.326591 14628 heartbeater.cc:503] Master 127.3.246.126:46725 was elected leader, sending a full tablet report...
W20260924 07:19:16.375308 14629 tablet.cc:2369] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:19:16.467007 14495 tablet.cc:2369] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:19:16.532706 14763 tablet.cc:2369] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:19:16.661327 14782 raft_consensus.cc:482] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:19:16.661463 14782 raft_consensus.cc:504] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d01d73deda449d9fb0ac770c395bba" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41957 } } peers { permanent_uuid: "3c194d23ab4040859716cc25356ee71e" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45731 } } peers { permanent_uuid: "cc0f97beae4d4ab0a0cc533188a7ee77" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41695 } }
I20260924 07:19:16.661900 14782 leader_election.cc:288] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9d01d73deda449d9fb0ac770c395bba (127.3.246.66:41957), cc0f97beae4d4ab0a0cc533188a7ee77 (127.3.246.67:41695)
I20260924 07:19:16.665151 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.662093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:41957 (local address 127.3.246.65:40683)
0924 07:19:16.662275 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:16.662293 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:16.662377 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:16.662556 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:16.662557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:16.662559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:16.662635 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.662642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.663314 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.663315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.664132 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.664134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.664874 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.664875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.664930 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.664934 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:19:16.665000 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:19:16.665042 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":1,"thread_start_us":61,"threads_started":1}
I20260924 07:19:16.665162 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.662209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:40683 (local address 127.3.246.66:41957)
0924 07:19:16.662261 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:16.662264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:16.662362 (+ 98us) server_negotiation.cc:407] Connection header received
0924 07:19:16.662468 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:16.662469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:16.662480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:16.662552 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:16.662713 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.663204 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.664229 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.664792 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.664869 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.664877 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:16.665019 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 07:19:16.665063 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:19:16.665261 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.662203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:42511 (local address 127.3.246.67:41695)
0924 07:19:16.662250 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:16.662253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:16.662409 (+ 156us) server_negotiation.cc:407] Connection header received
0924 07:19:16.662483 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:16.662484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:16.662493 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:16.662564 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:16.662734 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.663200 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.664364 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.664927 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.665017 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.665025 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:16.665145 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 07:19:16.665188 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:19:16.665261 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.662144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:41695 (local address 127.3.246.65:42511)
0924 07:19:16.662322 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:16.662331 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:16.662426 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:16.662567 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:16.662568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:16.662570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:16.662657 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.662659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.663282 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.663283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.664133 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:16.664135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.665013 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.665014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:16.665073 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.665076 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:19:16.665129 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:19:16.665161 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":14,"thread_start_us":62,"threads_started":1}
I20260924 07:19:16.665912 14582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3f143efa774b2dabfceaf096bbe3b2" candidate_uuid: "3c194d23ab4040859716cc25356ee71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d01d73deda449d9fb0ac770c395bba" is_pre_election: true
I20260924 07:19:16.665933 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3f143efa774b2dabfceaf096bbe3b2" candidate_uuid: "3c194d23ab4040859716cc25356ee71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc0f97beae4d4ab0a0cc533188a7ee77" is_pre_election: true
I20260924 07:19:16.666082 14582 raft_consensus.cc:2454] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c194d23ab4040859716cc25356ee71e in term 0.
I20260924 07:19:16.666082 14716 raft_consensus.cc:2454] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c194d23ab4040859716cc25356ee71e in term 0.
I20260924 07:19:16.666306 14382 leader_election.cc:302] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c194d23ab4040859716cc25356ee71e, d9d01d73deda449d9fb0ac770c395bba; no voters:
I20260924 07:19:16.666455 14782 raft_consensus.cc:2781] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:19:16.666509 14782 raft_consensus.cc:482] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:19:16.666554 14782 raft_consensus.cc:3037] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:16.667268 14782 raft_consensus.cc:504] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d01d73deda449d9fb0ac770c395bba" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41957 } } peers { permanent_uuid: "3c194d23ab4040859716cc25356ee71e" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45731 } } peers { permanent_uuid: "cc0f97beae4d4ab0a0cc533188a7ee77" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41695 } }
I20260924 07:19:16.667423 14782 leader_election.cc:288] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [CANDIDATE]: Term 1 election: Requested vote from peers d9d01d73deda449d9fb0ac770c395bba (127.3.246.66:41957), cc0f97beae4d4ab0a0cc533188a7ee77 (127.3.246.67:41695)
I20260924 07:19:16.667595 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3f143efa774b2dabfceaf096bbe3b2" candidate_uuid: "3c194d23ab4040859716cc25356ee71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc0f97beae4d4ab0a0cc533188a7ee77"
I20260924 07:19:16.667593 14582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3f143efa774b2dabfceaf096bbe3b2" candidate_uuid: "3c194d23ab4040859716cc25356ee71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d01d73deda449d9fb0ac770c395bba"
I20260924 07:19:16.667673 14716 raft_consensus.cc:3037] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:16.667673 14582 raft_consensus.cc:3037] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:16.668509 14716 raft_consensus.cc:2454] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c194d23ab4040859716cc25356ee71e in term 1.
I20260924 07:19:16.668509 14582 raft_consensus.cc:2454] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c194d23ab4040859716cc25356ee71e in term 1.
I20260924 07:19:16.668672 14382 leader_election.cc:302] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c194d23ab4040859716cc25356ee71e, d9d01d73deda449d9fb0ac770c395bba; no voters:
I20260924 07:19:16.668834 14782 raft_consensus.cc:2781] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:19:16.669011 14782 raft_consensus.cc:686] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [term 1 LEADER]: Becoming Leader. State: Replica: 3c194d23ab4040859716cc25356ee71e, State: Running, Role: LEADER
I20260924 07:19:16.669111 14782 consensus_queue.cc:237] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d01d73deda449d9fb0ac770c395bba" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41957 } } peers { permanent_uuid: "3c194d23ab4040859716cc25356ee71e" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45731 } } peers { permanent_uuid: "cc0f97beae4d4ab0a0cc533188a7ee77" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41695 } }
I20260924 07:19:16.669956 14301 catalog_manager.cc:5495] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c194d23ab4040859716cc25356ee71e (127.3.246.65). New cstate: current_term: 1 leader_uuid: "3c194d23ab4040859716cc25356ee71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d01d73deda449d9fb0ac770c395bba" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c194d23ab4040859716cc25356ee71e" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc0f97beae4d4ab0a0cc533188a7ee77" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41695 } health_report { overall_health: UNKNOWN } } }
I20260924 07:19:16.756697 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.754084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.3.246.65:45731)
0924 07:19:16.754151 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:16.754153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:16.754162 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:19:16.754237 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:16.754238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:16.754256 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:16.754313 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:19:16.754447 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.754928 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.755673 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:16.756054 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:16.756127 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:16.756324 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:19:16.756423 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:19:16.756489 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:16.756568 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 07:19:16.756586 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:19:16.758728 14582 raft_consensus.cc:1261] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba [term 1 FOLLOWER]: Refusing update from remote peer 3c194d23ab4040859716cc25356ee71e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:19:16.758733 14716 raft_consensus.cc:1261] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77 [term 1 FOLLOWER]: Refusing update from remote peer 3c194d23ab4040859716cc25356ee71e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:19:16.759105 14782 consensus_queue.cc:1035] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc0f97beae4d4ab0a0cc533188a7ee77" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:19:16.759295 14787 consensus_queue.cc:1035] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d01d73deda449d9fb0ac770c395bba" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:19:16.761767 14791 mvcc.cc:205] Tried to move back new op lower bound from 7332799925280768000 to 7332799924917379072. Current Snapshot: MvccSnapshot[applied={T|T < 7332799925280768000}]
I20260924 07:19:16.764668 14301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:19:16.764789 14301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:19:16.765558 14301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13577bcafc814e588e981f36c746dfb0: Sending DeleteTablet for 3 replicas of tablet ff3f143efa774b2dabfceaf096bbe3b2
I20260924 07:19:16.765924 14428 tablet_service.cc:1514] Processing DeleteTablet for tablet ff3f143efa774b2dabfceaf096bbe3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:19:16 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:56078
I20260924 07:19:16.765926 14562 tablet_service.cc:1514] Processing DeleteTablet for tablet ff3f143efa774b2dabfceaf096bbe3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:19:16 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:48840
I20260924 07:19:16.765972 14696 tablet_service.cc:1514] Processing DeleteTablet for tablet ff3f143efa774b2dabfceaf096bbe3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:19:16 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:55042
I20260924 07:19:16.766250 14800 tablet_replica.cc:330] T ff3f143efa774b2dabfceaf096bbe3b2 P 3c194d23ab4040859716cc25356ee71e: stopping tablet replica
I20260924 07:19:16.766245 14801 tablet_replica.cc:330] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba: stopping tablet replica
I20260924 07:19:16.766340 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 14365
I20260924 07:19:16.770345 14801 raft_consensus.cc:2229] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:19:16.770473 14801 raft_consensus.cc:2258] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:19:16.770923 14802 tablet_replica.cc:330] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77: stopping tablet replica
W20260924 07:19:16.770990 14288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:45731 (error 108)
I20260924 07:19:16.771122 14802 raft_consensus.cc:2229] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:19:16.771206 14801 ts_tablet_manager.cc:1905] T ff3f143efa774b2dabfceaf096bbe3b2 P d9d01d73deda449d9fb0ac770c395bba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:19:16.771281 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 14499
I20260924 07:19:16.771338 14802 raft_consensus.cc:2258] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:19:16.771996 14802 ts_tablet_manager.cc:1905] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:19:16.773547 14802 ts_tablet_manager.cc:1918] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:19:16.773615 14802 log.cc:1196] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/ff3f143efa774b2dabfceaf096bbe3b2
I20260924 07:19:16.773902 14802 ts_tablet_manager.cc:1939] T ff3f143efa774b2dabfceaf096bbe3b2 P cc0f97beae4d4ab0a0cc533188a7ee77: Deleting consensus metadata
I20260924 07:19:16.773936 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.773791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:45731 (local address 127.0.0.1:56104)
0924 07:19:16.773850 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:16.773893 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:45731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 07:19:16.774164 14288 catalog_manager.cc:4584] TS 3c194d23ab4040859716cc25356ee71e (127.3.246.65:45731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff3f143efa774b2dabfceaf096bbe3b2: Network error: Client connection negotiation failed: client connection to 127.3.246.65:45731: connect: Connection refused (error 111)
I20260924 07:19:16.774379 14290 catalog_manager.cc:4841] TS cc0f97beae4d4ab0a0cc533188a7ee77 (127.3.246.67:41695): tablet ff3f143efa774b2dabfceaf096bbe3b2 (table test-table [id=b3dcffdb56e24accb86d843d3a586d92]) successfully deleted
I20260924 07:19:16.777254 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:16.777134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:41957 (local address 127.0.0.1:48844)
0924 07:19:16.777190 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:16.777222 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.66:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 07:19:16.777328 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 14633
I20260924 07:19:16.782799 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 14271
2026-09-24T07:19:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2276 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:19:16 dist-test-slave-rc1w krb5kdc[14806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:19:16 dist-test-slave-rc1w krb5kdc[14806](info): set up 2 sockets
Sep 24 07:19:16 dist-test-slave-rc1w krb5kdc[14806](info): commencing operation
krb5kdc: starting...
W20260924 07:19:17.822732 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:19:17 dist-test-slave-rc1w krb5kdc[14806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:19:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:19:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/rangeradmin_127.3.246.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/rangeradmin_127.3.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/rangerlookup_127.3.246.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/rangerlookup_127.3.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/HTTP_127.3.246.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/HTTP_127.3.246.84@KRBTEST.COM.keytab.
I20260924 07:19:17.988547 4057 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwu82JT/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/postgres -l logfile start
2026-09-24 07:19:20.608 UTC [14836] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 07:19:20.608 UTC [14836] LOG: listening on IPv4 address "127.3.246.84", port 37107
2026-09-24 07:19:20.612 UTC [14836] LOG: listening on Unix socket "/tmp/.s.PGSQL.37107"
2026-09-24 07:19:20.648 UTC [14839] LOG: database system was shut down at 2026-09-24 07:19:19 UTC
2026-09-24 07:19:20.652 UTC [14836] LOG: database system is ready to accept connections
I20260924 07:19:21.586750 4057 mini_postgres.cc:96] Postgres bound to 37107
2026-09-24 07:19:21.592 UTC [14847] FATAL: database "slave" does not exist
127.3.246.84:37107 - accepting connections
I20260924 07:19:21.593276 4057 mini_ranger.cc:161] Starting Ranger...
I20260924 07:19:21.607736 4057 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 07:19:22.126087 4057 mini_ranger.cc:87] Created ranger Postgres database
I20260924 07:19:22.126183 4057 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-admin
2026-09-24 07:19:22,400 [I] DB FLAVOR :POSTGRES
2026-09-24 07:19:22,400 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:19:22,400 [I] Checking connection..
2026-09-24 07:19:22,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 07:19:22,657 [I] Checking connection passed.
2026-09-24 07:19:22,657 [I] --------- Verifying version history table ---------
2026-09-24 07:19:22,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:19:22,891 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:19:22,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:19:23,152 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:19:23,152 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 07:19:23,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 07:19:23.410 UTC [14911] WARNING: there is no transaction in progress
2026-09-24 07:19:23,416 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 07:19:23,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:19:23,665 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 07:19:23,665 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:19:23,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 07:19:23,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 07:19:24,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3m09.c.gcp-upstream.internal','N') ;"
2026-09-24 07:19:24,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:19:24,641 [I] Table x_portal_user does not exist in database ranger
2026-09-24 07:19:24,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:19:24,889 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 07:19:24,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:19:25,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:19:25,383 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 07:19:25,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 07:19:26.314 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.330 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.346 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.362 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.378 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.395 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.449 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.457 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.472 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.482 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.492 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.508 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.517 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.524 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.531 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:26.538 UTC [15019] WARNING: there is no transaction in progress
2026-09-24 07:19:27,178 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 07:19:27,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 07:19:27,418 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 07:19:27,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:19:27,653 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 07:19:27,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:19:27,911 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 07:19:27,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:19:28,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:19:28,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3m09.c.gcp-upstream.internal';"
2026-09-24 07:19:28,601 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 07:19:28,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 07:19:28,856 [I] DB_PATCHES have already been applied
I20260924 07:19:28.861852 4057 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwu82JT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 07:19:28.861917 4057 mini_ranger.cc:193] Using host: 127.3.246.84
I20260924 07:19:28.864809 4057 mini_ranger.cc:227] Ranger admin URL: http://127.3.246.84:51257
Sep 24, 2026 7:19:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:19:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:19:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:19:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:19:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:19:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:19:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 07:19:29.977916 4057 mini_ranger.cc:160] Time spent starting Ranger: real 8.385s user 0.003s sys 0.006s
Sep 24, 2026 7:19:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:19:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 7:19:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:19:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:19:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:19:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:19:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:19:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:19:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:19:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:19:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:19:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:19:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:19:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:19:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:19:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:19:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:19:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 07:19:50.356549 4057 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/rangerkms_127.3.246.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/rangerkms_127.3.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.246.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.246.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.246.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/HTTP_127.3.246.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.246.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/HTTP_127.3.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 07:19:50 dist-test-slave-rc1w krb5kdc[14806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234390, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 07:19:50.453097 4057 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 07:19:50.463543 4057 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 07:19:51.471050 4057 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 07:19:51.471145 4057 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-kms
I20260924 07:19:51.471170 4057 mini_ranger_kms.cc:227] Using postgres at 127.3.246.84:37107
2026-09-24 07:19:51,646 [I] DB FLAVOR :POSTGRES
2026-09-24 07:19:51,646 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:19:51,646 [I] Checking connection
2026-09-24 07:19:51,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 07:19:51,918 [I] connection success
2026-09-24 07:19:51,919 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 07:19:51,919 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 07:19:51,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 07:19:52,166 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 07:19:52,166 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:19:52,166 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 07:19:52,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:37107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 07:19:52,446 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 07:19:52.919783 4057 mini_ranger_kms.cc:310] Created kms service
I20260924 07:19:53.131448 4057 mini_ranger_kms.cc:326] Created kudu user
I20260924 07:19:53.176891 4057 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 07:19:53 dist-test-slave-rc1w krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234390, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:19:53.584508 4057 mini_ranger_kms.cc:384] Added ranger policy
I20260924 07:19:53.584666 4057 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 07:19:53.584689 4057 mini_ranger_kms.cc:240] Using host: 127.3.246.84
I20260924 07:19:53.587805 4057 mini_ranger_kms.cc:276] Ranger KMS PID: 15199
I20260924 07:19:53.587920 4057 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.246.84:49759
Sep 24, 2026 7:19:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:19:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:19:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:19:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:19:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:19:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
log4j:WARN getHostname()
java.net.UnknownHostException: dist-test-slave-3m09.c.gcp-upstream.internal: dist-test-slave-3m09.c.gcp-upstream.internal: Name or service not known
at java.net.InetAddress.getLocalHost(InetAddress.java:1432)
at org.apache.ranger.audit.provider.MiscUtil.initLocalHost(MiscUtil.java:843)
at org.apache.ranger.audit.provider.MiscUtil.<clinit>(MiscUtil.java:100)
at org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer.authWithKerberos(RangerKmsAuthorizer.java:150)
at org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer.<init>(RangerKmsAuthorizer.java:126)
at org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer.<init>(RangerKmsAuthorizer.java:154)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer.init(RangerKmsAuthorizer.java:71)
at org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer.<init>(RangerKmsAuthorizer.java:51)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.crypto.key.kms.server.KMSWebApp.getAcls(KMSWebApp.java:240)
at org.apache.hadoop.crypto.key.kms.server.KMSWebApp.contextInitialized(KMSWebApp.java:139)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5128)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5653)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1689)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1679)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.net.UnknownHostException: dist-test-slave-3m09.c.gcp-upstream.internal: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:867)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:815)
at java.net.InetAddress.getAllByName0(InetAddress.java:1291)
at java.net.InetAddress.getLocalHost(InetAddress.java:1427)
... 28 more
Sep 24 07:19:55 dist-test-slave-rc1w krb5kdc[14806](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790234395, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.246.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:55.826772 4057 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.374s user 0.001s sys 0.157s
I20260924 07:19:55.826865 4057 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 07:19:55.826956 4057 mini_ranger_kms.cc:401] 127.3.246.84:49759/kms/v1/keys
Sep 24 07:19:56 dist-test-slave-rc1w krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234395, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.246.84@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
Sep 24 07:19:56 dist-test-slave-rc1w krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234395, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.246.84@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
Sep 24 07:19:58 dist-test-slave-rc1w krb5kdc[14806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:19:58.696919 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:41625
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:41625
--ranger_config_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.246.84:49759/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:19:58.780633 15250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:58.780870 15250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:58.780905 15250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:58.782101 15250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:19:58.782161 15250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:58.782183 15250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:58.782204 15250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:19:58.782258 15250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:19:58.783771 15250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.246.84:49759/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:41625
--ranger_config_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:41625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:58.784034 15250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:58.784257 15250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:58.790380 15255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:58.790567 15258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:58.790534 15250 server_base.cc:1034] running on GCE node
W20260924 07:19:58.790401 15256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:58.790988 15250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:58.791328 15250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:58.792513 15250 hybrid_clock.cc:648] HybridClock initialized: now 1790234398792459 us; error 71 us; skew 500 ppm
Sep 24 07:19:58 dist-test-slave-rc1w krb5kdc[14806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234398, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:58.802431 15250 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:19:58.802935 15250 webserver.cc:460] Webserver started at http://127.3.246.126:35693/ using document root <none> and password file <none>
I20260924 07:19:58.803184 15250 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:58.803222 15250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:58.803356 15250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:19:58 dist-test-slave-rc1w krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234398, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:19:58.863162 15250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "0e70a50d8f7b45c0a2283e4bf6c2973f"
format_stamp: "Formatted at 2026-09-24 07:19:58 on dist-test-slave-rc1w"
server_key: "2d65223d20e073e5af8427c417f8c984"
server_key_iv: "526771397b0661ceda8cfc4d021354b3"
server_key_version: "kuduclusterkey@0"
I20260924 07:19:58.863570 15250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "0e70a50d8f7b45c0a2283e4bf6c2973f"
format_stamp: "Formatted at 2026-09-24 07:19:58 on dist-test-slave-rc1w"
server_key: "2d65223d20e073e5af8427c417f8c984"
server_key_iv: "526771397b0661ceda8cfc4d021354b3"
server_key_version: "kuduclusterkey@0"
I20260924 07:19:58.865046 15250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:19:58.879873 15265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:58.880059 15250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:19:58.880167 15250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "0e70a50d8f7b45c0a2283e4bf6c2973f"
format_stamp: "Formatted at 2026-09-24 07:19:58 on dist-test-slave-rc1w"
server_key: "2d65223d20e073e5af8427c417f8c984"
server_key_iv: "526771397b0661ceda8cfc4d021354b3"
server_key_version: "kuduclusterkey@0"
I20260924 07:19:58.880273 15250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:58.898130 15250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:58.900241 15250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:58.900400 15250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:58.905769 15250 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:41625
I20260924 07:19:58.905800 15317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:41625 every 8 connection(s)
I20260924 07:19:58.906143 15250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:19:58.906814 15318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:58.909003 15318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f: Bootstrap starting.
I20260924 07:19:58.909574 15318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:58.909889 15318 log.cc:824] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:58.910542 15318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f: No bootstrap required, opened a new log
I20260924 07:19:58.912537 15318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e70a50d8f7b45c0a2283e4bf6c2973f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41625 } }
I20260924 07:19:58.912706 15318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:58.912810 15318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e70a50d8f7b45c0a2283e4bf6c2973f, State: Initialized, Role: FOLLOWER
I20260924 07:19:58.912972 15318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e70a50d8f7b45c0a2283e4bf6c2973f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41625 } }
I20260924 07:19:58.913058 15318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:19:58.913100 15318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:19:58.913208 15318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:58.913347 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 15250
I20260924 07:19:58.913442 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:19:58.913633 4057 external_mini_cluster.cc:1428] Setting key 074f08170aca59cf85ae0dee3dd2e3ae
I20260924 07:19:58.914381 15318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e70a50d8f7b45c0a2283e4bf6c2973f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41625 } }
I20260924 07:19:58.914512 15318 leader_election.cc:302] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e70a50d8f7b45c0a2283e4bf6c2973f; no voters:
I20260924 07:19:58.914686 15318 leader_election.cc:288] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:19:58.914716 15323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:19:58.914925 15323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [term 1 LEADER]: Becoming Leader. State: Replica: 0e70a50d8f7b45c0a2283e4bf6c2973f, State: Running, Role: LEADER
I20260924 07:19:58.915014 15318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:19:58.915050 15323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e70a50d8f7b45c0a2283e4bf6c2973f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41625 } }
I20260924 07:19:58.915453 15323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e70a50d8f7b45c0a2283e4bf6c2973f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e70a50d8f7b45c0a2283e4bf6c2973f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41625 } } }
I20260924 07:19:58.915637 15323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:58.915822 15323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e70a50d8f7b45c0a2283e4bf6c2973f. Latest consensus state: current_term: 1 leader_uuid: "0e70a50d8f7b45c0a2283e4bf6c2973f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e70a50d8f7b45c0a2283e4bf6c2973f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41625 } } }
I20260924 07:19:58.915934 15323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:58.916112 15332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:19:58.916671 15332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:19:58.916996 15318 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:19:58.922152 15340 mvcc.cc:205] Tried to move back new op lower bound from 7332800097980899328 to 7332800097956556800. Current Snapshot: MvccSnapshot[applied={T|T < 7332800097980899328}]
I20260924 07:19:58.923125 15332 catalog_manager.cc:1347] Generated new cluster ID: baaa935b5ecc4abea94b4daa9a92dbd6
Sep 24 07:19:58 dist-test-slave-rc1w krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:19:58.923192 15332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:19:58.928067 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:58.914161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.3.246.126:41625)
0924 07:19:58.914361 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:58.914371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:58.914402 (+ 31us) server_negotiation.cc:407] Connection header received
0924 07:19:58.914521 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:58.914528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:58.914591 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:58.914746 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:58.915621 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:58.917036 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:58.918680 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:58.920627 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:58.920801 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:58.923881 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:58.923885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:58.923885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:58.923895 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:58.926046 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:58.926414 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:58.926417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:58.926417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:58.926518 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:58.926708 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:58.926708 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:58.926708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:58.927007 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:58.927152 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:58.927612 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 07:19:58.927791 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260924 07:19:58.930064 15332 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:19:58.931128 15332 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:19:58.939800 15332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f: Generated new TSK 0
I20260924 07:19:58.940049 15332 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:19:59.685969 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.246.84:49759/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:41625
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:19:59.784152 15364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:59.784402 15364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:59.784446 15364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:59.785797 15364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:59.785867 15364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:59.785928 15364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:19:59.787503 15364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.246.84:49759/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:41625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:19:59.787737 15364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:59.787953 15364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:59.794055 15370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:59.794183 15369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:59.794289 15364 server_base.cc:1034] running on GCE node
W20260924 07:19:59.794034 15372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:59.797890 15364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:59.798175 15364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:59.799352 15364 hybrid_clock.cc:648] HybridClock initialized: now 1790234399799327 us; error 41 us; skew 500 ppm
Sep 24 07:19:59 dist-test-slave-rc1w krb5kdc[14806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234399, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:59.802619 15364 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:19:59.803138 15364 webserver.cc:460] Webserver started at http://127.3.246.65:42885/ using document root <none> and password file <none>
I20260924 07:19:59.803469 15364 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:59.803544 15364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:59.803727 15364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:19:59 dist-test-slave-rc1w krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234399, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:19:59.816792 15364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "55aaec9ea3494fe3818d588d845b4837"
format_stamp: "Formatted at 2026-09-24 07:19:59 on dist-test-slave-rc1w"
server_key: "404ea717c88a61a9aa35794c0c57b16d"
server_key_iv: "409bbab94a613d80c1b607f563a01b43"
server_key_version: "kuduclusterkey@0"
I20260924 07:19:59.817194 15364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "55aaec9ea3494fe3818d588d845b4837"
format_stamp: "Formatted at 2026-09-24 07:19:59 on dist-test-slave-rc1w"
server_key: "404ea717c88a61a9aa35794c0c57b16d"
server_key_iv: "409bbab94a613d80c1b607f563a01b43"
server_key_version: "kuduclusterkey@0"
I20260924 07:19:59.824396 15364 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.003s
I20260924 07:19:59.837148 15381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:59.837359 15364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:19:59.837441 15364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "55aaec9ea3494fe3818d588d845b4837"
format_stamp: "Formatted at 2026-09-24 07:19:59 on dist-test-slave-rc1w"
server_key: "404ea717c88a61a9aa35794c0c57b16d"
server_key_iv: "409bbab94a613d80c1b607f563a01b43"
server_key_version: "kuduclusterkey@0"
I20260924 07:19:59.837527 15364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:59.858233 15364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:59.860452 15364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:59.860646 15364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:59.860917 15364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:59.861327 15364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:59.861404 15364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:59.861450 15364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:59.861471 15364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:59.869573 15364 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:45823
I20260924 07:19:59.869661 15494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:45823 every 8 connection(s)
I20260924 07:19:59.870008 15364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:19:59.872738 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 15364
I20260924 07:19:59.872845 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:19:59.873014 4057 external_mini_cluster.cc:1428] Setting key 6a648d3de2a04b83801f5366267d9b47
Sep 24 07:19:59 dist-test-slave-rc1w krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234399, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
I20260924 07:19:59.887190 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:59.870799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:38775 (local address 127.3.246.126:41625)
0924 07:19:59.871825 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:59.871827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:59.871835 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:19:59.871863 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:59.871863 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:59.871877 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:59.871947 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:59.872678 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:59.873469 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:59.876562 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:59.876881 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:59.876924 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:59.883383 (+ 6459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:59.883385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:59.883386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:59.883394 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:59.885393 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:59.886035 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:59.886036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:59.886036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:59.886102 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:59.886394 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:59.886395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:59.886395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:59.886536 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:59.886589 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:59.886998 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 07:19:59.887088 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":82,"threads_started":1}
I20260924 07:19:59.887733 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:59.870768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:41625 (local address 127.3.246.65:38775)
0924 07:19:59.870959 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:59.871008 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:59.871638 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:59.872050 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:59.872055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:59.872290 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:59.872553 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:59.872559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:59.873578 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:59.873579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:59.876452 (+ 2873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:59.876455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:59.877014 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:59.877015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:59.877153 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:59.881485 (+ 4332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:59.881496 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:59.883250 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:59.885522 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:59.885529 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:59.885532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:59.885935 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:59.886178 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:59.886179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:59.886180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:59.886316 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:59.886623 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:59.886629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:59.886826 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 07:19:59.887328 (+ 502us) client_negotiation.cc:240] Negotiation successful
0924 07:19:59.887499 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260924 07:19:59.888321 15495 heartbeater.cc:347] Connected to a master server at 127.3.246.126:41625
I20260924 07:19:59.888458 15495 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:59.888679 15495 heartbeater.cc:511] Master 127.3.246.126:41625 requested a full tablet report, sending...
I20260924 07:19:59.889228 15278 ts_manager.cc:194] Registered new tserver with Master: 55aaec9ea3494fe3818d588d845b4837 (127.3.246.65:45823)
I20260924 07:19:59.890172 15278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:38775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:19:59.916903 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.246.84:49759/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:41625
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:19:59 dist-test-slave-rc1w krb5kdc[14806](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790234399, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 07:20:00.016639 15503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:00.016981 15503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:00.017066 15503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:00.018899 15503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:00.019012 15503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:00.019105 15503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:20:00.021289 15503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.246.84:49759/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:41625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:00.021764 15503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:00.022092 15503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:00.027464 15508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:00.028057 15509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:00.028260 15511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:00.028748 15503 server_base.cc:1034] running on GCE node
I20260924 07:20:00.028961 15503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:00.029196 15503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:00.030357 15503 hybrid_clock.cc:648] HybridClock initialized: now 1790234400030335 us; error 35 us; skew 500 ppm
Sep 24 07:20:00 dist-test-slave-rc1w krb5kdc[14806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:00.033771 15503 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:20:00.034273 15503 webserver.cc:460] Webserver started at http://127.3.246.66:37579/ using document root <none> and password file <none>
I20260924 07:20:00.034534 15503 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:00.034592 15503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:00.034772 15503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:20:00 dist-test-slave-rc1w krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:00.049373 15503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "d60c8c49796942ca8c9d371629e4ecfe"
format_stamp: "Formatted at 2026-09-24 07:20:00 on dist-test-slave-rc1w"
server_key: "a8e69fb52ba56b7e00fe8c3d46a3c500"
server_key_iv: "97a8e2713d733d2c5e2adc26ffbab6bb"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:00.049945 15503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "d60c8c49796942ca8c9d371629e4ecfe"
format_stamp: "Formatted at 2026-09-24 07:20:00 on dist-test-slave-rc1w"
server_key: "a8e69fb52ba56b7e00fe8c3d46a3c500"
server_key_iv: "97a8e2713d733d2c5e2adc26ffbab6bb"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:00.051985 15503 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:20:00.071990 15519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:00.072237 15503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:00.072340 15503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "d60c8c49796942ca8c9d371629e4ecfe"
format_stamp: "Formatted at 2026-09-24 07:20:00 on dist-test-slave-rc1w"
server_key: "a8e69fb52ba56b7e00fe8c3d46a3c500"
server_key_iv: "97a8e2713d733d2c5e2adc26ffbab6bb"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:00.072429 15503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:00.096910 15503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:00.100277 15503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:00.100428 15503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:00.100653 15503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:00.101065 15503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:00.101114 15503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:00.101145 15503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:00.101218 15503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:00.108753 15503 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:44921
I20260924 07:20:00.109273 15503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:20:00.117173 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 15503
I20260924 07:20:00.117286 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:20:00.117498 4057 external_mini_cluster.cc:1428] Setting key 82ccb59f018f41542ad4a6176c89ef2a
Sep 24 07:20:00 dist-test-slave-rc1w krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:00.120756 15632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:44921 every 8 connection(s)
I20260924 07:20:00.121974 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.110132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:43917 (local address 127.3.246.126:41625)
0924 07:20:00.110187 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:00.110189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:00.110433 (+ 244us) server_negotiation.cc:407] Connection header received
0924 07:20:00.111166 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:00.111167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:00.111185 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:00.111257 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:00.112017 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.113204 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.115138 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.115500 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.115585 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.118393 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:00.118395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:00.118395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:00.118403 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:00.120139 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:00.120656 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:00.120658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:00.120658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:00.120725 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:00.121018 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:00.121019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:00.121019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:00.121210 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:00.121306 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:00.121711 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 07:20:00.121819 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:20:00.122036 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.110123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:41625 (local address 127.3.246.66:43917)
0924 07:20:00.110288 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:00.110338 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:00.110982 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:00.111359 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:00.111363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:00.111615 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:00.111890 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.111897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.113333 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.113334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.114990 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.114994 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.115604 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.115605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.115751 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.116590 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:00.116601 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:00.118274 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:00.120247 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:00.120255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:00.120259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:00.120560 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:00.120804 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:00.120805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:00.120805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:00.120930 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:00.121304 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:00.121311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:00.121482 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 07:20:00.121605 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 07:20:00.121797 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260924 07:20:00.122862 15633 heartbeater.cc:347] Connected to a master server at 127.3.246.126:41625
I20260924 07:20:00.123034 15633 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:00.123307 15633 heartbeater.cc:511] Master 127.3.246.126:41625 requested a full tablet report, sending...
I20260924 07:20:00.123796 15278 ts_manager.cc:194] Registered new tserver with Master: d60c8c49796942ca8c9d371629e4ecfe (127.3.246.66:44921)
I20260924 07:20:00.124300 15278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:43917
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:20:00.155054 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.246.84:49759/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:41625
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:00.252003 15638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:00.252249 15638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:00.252286 15638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:00.253530 15638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:00.253603 15638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:00.253648 15638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:20:00.255410 15638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.246.84:49759/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:41625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:00.255885 15638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:00.256218 15638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:00.261745 15644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:00.261873 15646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:00.261752 15643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:00.263126 15638 server_base.cc:1034] running on GCE node
I20260924 07:20:00.263347 15638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:00.263666 15638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:00.266954 15638 hybrid_clock.cc:648] HybridClock initialized: now 1790234400266915 us; error 50 us; skew 500 ppm
Sep 24 07:20:00 dist-test-slave-rc1w krb5kdc[14806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:00.270242 15638 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:20:00.270794 15638 webserver.cc:460] Webserver started at http://127.3.246.67:40201/ using document root <none> and password file <none>
I20260924 07:20:00.271064 15638 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:00.271124 15638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:00.271265 15638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:20:00 dist-test-slave-rc1w krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:00.284467 15638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "672eb8a049864e7cb2df4370d1db33cf"
format_stamp: "Formatted at 2026-09-24 07:20:00 on dist-test-slave-rc1w"
server_key: "96923844e9d46056ba22342623ed1fe0"
server_key_iv: "4f5e9df3d142d0cc4a4578516b75f6d9"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:00.284941 15638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "672eb8a049864e7cb2df4370d1db33cf"
format_stamp: "Formatted at 2026-09-24 07:20:00 on dist-test-slave-rc1w"
server_key: "96923844e9d46056ba22342623ed1fe0"
server_key_iv: "4f5e9df3d142d0cc4a4578516b75f6d9"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:00.290109 15638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 07:20:00.317935 15653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:00.318182 15638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:20:00.318300 15638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "672eb8a049864e7cb2df4370d1db33cf"
format_stamp: "Formatted at 2026-09-24 07:20:00 on dist-test-slave-rc1w"
server_key: "96923844e9d46056ba22342623ed1fe0"
server_key_iv: "4f5e9df3d142d0cc4a4578516b75f6d9"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:00.318480 15638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:00.339571 15638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:00.345052 15638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:00.345293 15638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:00.345564 15638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:00.345999 15638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:00.346092 15638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:00.346164 15638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:00.346220 15638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:00.357115 15638 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:42989
I20260924 07:20:00.357857 15638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:20:00.358825 15766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:42989 every 8 connection(s)
I20260924 07:20:00.361552 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 15638
I20260924 07:20:00.361734 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:20:00.361939 4057 external_mini_cluster.cc:1428] Setting key bcb8126ec3fe4a7c90081e0c09c735ca
Sep 24 07:20:00 dist-test-slave-rc1w krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:00.376482 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.358897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:41253 (local address 127.3.246.126:41625)
0924 07:20:00.358944 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:00.358946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:00.366044 (+ 7098us) server_negotiation.cc:407] Connection header received
0924 07:20:00.366791 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:00.366792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:00.366807 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:00.366856 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:00.368162 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.369060 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.370177 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.370454 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.370524 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.372897 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:00.372898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:00.372898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:00.372906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:00.374762 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:00.375338 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:00.375339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:00.375340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:00.375408 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:00.375696 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:00.375697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:00.375697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:00.375862 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:00.375924 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:00.376306 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 07:20:00.376386 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:20:00.376889 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.358885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:41625 (local address 127.3.246.67:41253)
0924 07:20:00.365868 (+ 6983us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:00.365942 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:00.366637 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:00.366937 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:00.366946 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:00.367724 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:00.368028 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.368036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.369188 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.369189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.370086 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.370088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.370598 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.370599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.370793 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.371388 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:00.371398 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:00.372799 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:00.374881 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:00.374888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:00.374891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:00.375211 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:00.375480 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:00.375481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:00.375482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:00.375607 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:00.375915 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:00.375921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:00.376143 (+ 222us) client_negotiation.cc:769] Sending connection context
0924 07:20:00.376273 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 07:20:00.376425 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6919,"thread_start_us":88,"threads_started":1}
I20260924 07:20:00.377380 15767 heartbeater.cc:347] Connected to a master server at 127.3.246.126:41625
I20260924 07:20:00.377498 15767 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:00.377735 15767 heartbeater.cc:511] Master 127.3.246.126:41625 requested a full tablet report, sending...
I20260924 07:20:00.378211 15278 ts_manager.cc:194] Registered new tserver with Master: 672eb8a049864e7cb2df4370d1db33cf (127.3.246.67:42989)
I20260924 07:20:00.378605 15278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:41253
I20260924 07:20:00.384866 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:20:00.394605 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.386645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.3.246.126:41625)
0924 07:20:00.386691 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:00.386693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:00.387009 (+ 316us) server_negotiation.cc:407] Connection header received
0924 07:20:00.387382 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:00.387383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:00.387394 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:00.387483 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:00.388064 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.388808 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.389585 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.389871 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.389951 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.391541 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:00.391543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:00.391543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:00.391551 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:00.393292 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:00.393767 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:00.393768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:00.393769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:00.393828 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:00.394039 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:00.394040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:00.394041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:00.394166 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:00.394275 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:00.394408 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 07:20:00.394488 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:20:00.396533 15278 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:20:00.398764 15278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:20:00.408799 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.402008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:45823 (local address 127.0.0.1:59264)
0924 07:20:00.403783 (+ 1775us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:00.403848 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:00.403948 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:00.404222 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:00.404225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:00.404227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:00.404306 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.404308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.405618 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.405619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.407014 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.407017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.408472 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.408473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.408555 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.408571 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 07:20:00.408647 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:20:00.408708 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"thread_start_us":81,"threads_started":1}
I20260924 07:20:00.409394 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.402239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:44921 (local address 127.0.0.1:56444)
0924 07:20:00.404061 (+ 1822us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:00.404072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:00.404125 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:00.404381 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:00.404381 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:00.404382 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:00.404418 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.404419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.405995 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.405996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.407038 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.407040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.408757 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.408758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.408816 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.408820 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:20:00.409281 (+ 461us) client_negotiation.cc:240] Negotiation successful
0924 07:20:00.409327 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1808,"thread_start_us":82,"threads_started":1}
I20260924 07:20:00.409842 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.403159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.3.246.66:44921)
0924 07:20:00.403356 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:00.403362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:00.404180 (+ 818us) server_negotiation.cc:407] Connection header received
0924 07:20:00.404224 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:00.404226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:00.404240 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:00.404309 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:00.404488 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.405910 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.407151 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.408585 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.408650 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.408725 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:00.409714 (+ 989us) server_negotiation.cc:299] Negotiation successful
0924 07:20:00.409761 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":125,"threads_started":1}
I20260924 07:20:00.410557 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.402997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.3.246.65:45823)
0924 07:20:00.403666 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:00.403670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:00.403929 (+ 259us) server_negotiation.cc:407] Connection header received
0924 07:20:00.404026 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:00.404029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:00.404047 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:00.404125 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:00.404381 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.405520 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.407117 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.408306 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.408374 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.408439 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:00.410439 (+ 2000us) server_negotiation.cc:299] Negotiation successful
0924 07:20:00.410488 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":57,"threads_started":1}
I20260924 07:20:00.410815 15562 tablet_service.cc:1467] Processing CreateTablet for tablet 66cbe6271ea94a31978183b8d3d8860e (DEFAULT_TABLE table=test-table [id=f35fd0efdab3401cb2b8e631b41f0f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:00.411235 15562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cbe6271ea94a31978183b8d3d8860e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:00.411334 15429 tablet_service.cc:1467] Processing CreateTablet for tablet 66cbe6271ea94a31978183b8d3d8860e (DEFAULT_TABLE table=test-table [id=f35fd0efdab3401cb2b8e631b41f0f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:00.411638 15429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cbe6271ea94a31978183b8d3d8860e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:00.413076 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.402510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:42989 (local address 127.0.0.1:51080)
0924 07:20:00.408599 (+ 6089us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:00.408614 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:00.408693 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:00.409155 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:00.409156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:00.409158 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:00.409192 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.409193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.410414 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.410415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.411782 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.411785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.412838 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.412839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.412907 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.412912 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:20:00.412963 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:20:00.413006 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6059,"thread_start_us":62,"threads_started":1}
I20260924 07:20:00.413369 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.403378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.3.246.67:42989)
0924 07:20:00.403536 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:00.403540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:00.408917 (+ 5377us) server_negotiation.cc:407] Connection header received
0924 07:20:00.408971 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:00.408975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:00.408995 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:00.409075 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:00.409267 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.410341 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.411919 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.412639 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.412726 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.412800 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:00.413251 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 07:20:00.413293 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260924 07:20:00.413968 15701 tablet_service.cc:1467] Processing CreateTablet for tablet 66cbe6271ea94a31978183b8d3d8860e (DEFAULT_TABLE table=test-table [id=f35fd0efdab3401cb2b8e631b41f0f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:00.414274 15701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cbe6271ea94a31978183b8d3d8860e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:00.415087 15786 tablet_bootstrap.cc:492] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837: Bootstrap starting.
I20260924 07:20:00.415308 15785 tablet_bootstrap.cc:492] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe: Bootstrap starting.
I20260924 07:20:00.417199 15786 tablet_bootstrap.cc:654] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:00.417590 15786 log.cc:824] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:00.416836 15785 tablet_bootstrap.cc:654] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:00.418306 15785 log.cc:824] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:00.419015 15786 tablet_bootstrap.cc:492] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837: No bootstrap required, opened a new log
I20260924 07:20:00.419121 15785 tablet_bootstrap.cc:492] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe: No bootstrap required, opened a new log
I20260924 07:20:00.419132 15786 ts_tablet_manager.cc:1397] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 07:20:00.419186 15785 ts_tablet_manager.cc:1397] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 07:20:00.421736 15785 raft_consensus.cc:348] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55aaec9ea3494fe3818d588d845b4837" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45823 } } peers { permanent_uuid: "d60c8c49796942ca8c9d371629e4ecfe" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 44921 } } peers { permanent_uuid: "672eb8a049864e7cb2df4370d1db33cf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42989 } }
I20260924 07:20:00.421923 15785 raft_consensus.cc:374] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:00.421967 15785 raft_consensus.cc:729] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60c8c49796942ca8c9d371629e4ecfe, State: Initialized, Role: FOLLOWER
I20260924 07:20:00.422309 15786 raft_consensus.cc:348] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55aaec9ea3494fe3818d588d845b4837" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45823 } } peers { permanent_uuid: "d60c8c49796942ca8c9d371629e4ecfe" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 44921 } } peers { permanent_uuid: "672eb8a049864e7cb2df4370d1db33cf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42989 } }
I20260924 07:20:00.422459 15785 consensus_queue.cc:260] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55aaec9ea3494fe3818d588d845b4837" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45823 } } peers { permanent_uuid: "d60c8c49796942ca8c9d371629e4ecfe" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 44921 } } peers { permanent_uuid: "672eb8a049864e7cb2df4370d1db33cf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42989 } }
I20260924 07:20:00.422533 15786 raft_consensus.cc:374] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:00.422724 15786 raft_consensus.cc:729] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55aaec9ea3494fe3818d588d845b4837, State: Initialized, Role: FOLLOWER
I20260924 07:20:00.422752 15785 ts_tablet_manager.cc:1428] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 07:20:00.422890 15633 heartbeater.cc:503] Master 127.3.246.126:41625 was elected leader, sending a full tablet report...
I20260924 07:20:00.422921 15786 consensus_queue.cc:260] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55aaec9ea3494fe3818d588d845b4837" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45823 } } peers { permanent_uuid: "d60c8c49796942ca8c9d371629e4ecfe" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 44921 } } peers { permanent_uuid: "672eb8a049864e7cb2df4370d1db33cf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42989 } }
I20260924 07:20:00.423187 15786 ts_tablet_manager.cc:1428] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 07:20:00.423262 15495 heartbeater.cc:503] Master 127.3.246.126:41625 was elected leader, sending a full tablet report...
I20260924 07:20:00.423385 15787 tablet_bootstrap.cc:492] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf: Bootstrap starting.
I20260924 07:20:00.424458 15787 tablet_bootstrap.cc:654] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:00.424759 15787 log.cc:824] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:00.425526 15787 tablet_bootstrap.cc:492] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf: No bootstrap required, opened a new log
I20260924 07:20:00.425626 15787 ts_tablet_manager.cc:1397] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:00.426993 15787 raft_consensus.cc:348] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55aaec9ea3494fe3818d588d845b4837" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45823 } } peers { permanent_uuid: "d60c8c49796942ca8c9d371629e4ecfe" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 44921 } } peers { permanent_uuid: "672eb8a049864e7cb2df4370d1db33cf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42989 } }
I20260924 07:20:00.427146 15787 raft_consensus.cc:374] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:00.427191 15787 raft_consensus.cc:729] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672eb8a049864e7cb2df4370d1db33cf, State: Initialized, Role: FOLLOWER
I20260924 07:20:00.427541 15787 consensus_queue.cc:260] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55aaec9ea3494fe3818d588d845b4837" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45823 } } peers { permanent_uuid: "d60c8c49796942ca8c9d371629e4ecfe" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 44921 } } peers { permanent_uuid: "672eb8a049864e7cb2df4370d1db33cf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42989 } }
I20260924 07:20:00.427865 15767 heartbeater.cc:503] Master 127.3.246.126:41625 was elected leader, sending a full tablet report...
I20260924 07:20:00.427858 15787 ts_tablet_manager.cc:1428] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:00.434748 15790 raft_consensus.cc:482] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:00.434880 15790 raft_consensus.cc:504] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55aaec9ea3494fe3818d588d845b4837" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45823 } } peers { permanent_uuid: "d60c8c49796942ca8c9d371629e4ecfe" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 44921 } } peers { permanent_uuid: "672eb8a049864e7cb2df4370d1db33cf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42989 } }
I20260924 07:20:00.435312 15790 leader_election.cc:288] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55aaec9ea3494fe3818d588d845b4837 (127.3.246.65:45823), 672eb8a049864e7cb2df4370d1db33cf (127.3.246.67:42989)
I20260924 07:20:00.439303 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.435620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:51233 (local address 127.3.246.67:42989)
0924 07:20:00.435661 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:00.435663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:00.436077 (+ 414us) server_negotiation.cc:407] Connection header received
0924 07:20:00.436172 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:00.436173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:00.436184 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:00.436211 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:00.436401 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.437065 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.438375 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.438933 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.438983 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.438994 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:00.439202 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 07:20:00.439237 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:20:00.439304 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.435407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:45823 (local address 127.3.246.66:42757)
0924 07:20:00.435443 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:00.435455 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:00.435498 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:00.435947 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:00.435948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:00.435950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:00.436021 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.436023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.436991 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.436992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.438256 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.438260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.439038 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.439039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.439114 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.439143 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 07:20:00.439191 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 07:20:00.439231 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":12}
I20260924 07:20:00.439579 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.435771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:42757 (local address 127.3.246.65:45823)
0924 07:20:00.435825 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:00.435827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:00.435834 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:20:00.435865 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:00.435866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:00.435873 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:00.435942 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:00.436104 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.436904 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.438369 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.438937 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.439000 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.439007 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:00.439477 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 07:20:00.439522 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:20:00.439844 15449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbe6271ea94a31978183b8d3d8860e" candidate_uuid: "d60c8c49796942ca8c9d371629e4ecfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55aaec9ea3494fe3818d588d845b4837" is_pre_election: true
I20260924 07:20:00.439913 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.435530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:42989 (local address 127.3.246.66:51233)
0924 07:20:00.436020 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:00.436030 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:00.436101 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:00.436268 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:00.436269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:00.436270 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:00.436332 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.436333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.437165 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.437166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.438272 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:00.438274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.439047 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.439048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:00.439114 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.439118 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:20:00.439818 (+ 700us) client_negotiation.cc:240] Negotiation successful
0924 07:20:00.439850 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":129,"threads_started":1}
I20260924 07:20:00.440011 15449 raft_consensus.cc:2454] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d60c8c49796942ca8c9d371629e4ecfe in term 0.
I20260924 07:20:00.440251 15521 leader_election.cc:302] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55aaec9ea3494fe3818d588d845b4837, d60c8c49796942ca8c9d371629e4ecfe; no voters:
I20260924 07:20:00.440231 15721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbe6271ea94a31978183b8d3d8860e" candidate_uuid: "d60c8c49796942ca8c9d371629e4ecfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672eb8a049864e7cb2df4370d1db33cf" is_pre_election: true
I20260924 07:20:00.440363 15721 raft_consensus.cc:2454] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d60c8c49796942ca8c9d371629e4ecfe in term 0.
I20260924 07:20:00.440469 15790 raft_consensus.cc:2781] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:20:00.440563 15790 raft_consensus.cc:482] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:20:00.440627 15790 raft_consensus.cc:3037] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:00.441416 15790 raft_consensus.cc:504] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55aaec9ea3494fe3818d588d845b4837" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45823 } } peers { permanent_uuid: "d60c8c49796942ca8c9d371629e4ecfe" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 44921 } } peers { permanent_uuid: "672eb8a049864e7cb2df4370d1db33cf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42989 } }
I20260924 07:20:00.441825 15721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbe6271ea94a31978183b8d3d8860e" candidate_uuid: "d60c8c49796942ca8c9d371629e4ecfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672eb8a049864e7cb2df4370d1db33cf"
I20260924 07:20:00.441910 15721 raft_consensus.cc:3037] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:00.442054 15449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbe6271ea94a31978183b8d3d8860e" candidate_uuid: "d60c8c49796942ca8c9d371629e4ecfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55aaec9ea3494fe3818d588d845b4837"
I20260924 07:20:00.442127 15449 raft_consensus.cc:3037] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:00.442822 15721 raft_consensus.cc:2454] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d60c8c49796942ca8c9d371629e4ecfe in term 1.
I20260924 07:20:00.443027 15520 leader_election.cc:302] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672eb8a049864e7cb2df4370d1db33cf, d60c8c49796942ca8c9d371629e4ecfe; no voters:
I20260924 07:20:00.443037 15449 raft_consensus.cc:2454] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d60c8c49796942ca8c9d371629e4ecfe in term 1.
I20260924 07:20:00.443200 15790 leader_election.cc:288] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [CANDIDATE]: Term 1 election: Requested vote from peers 55aaec9ea3494fe3818d588d845b4837 (127.3.246.65:45823), 672eb8a049864e7cb2df4370d1db33cf (127.3.246.67:42989)
I20260924 07:20:00.443255 15790 raft_consensus.cc:2781] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:00.443300 15790 raft_consensus.cc:686] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [term 1 LEADER]: Becoming Leader. State: Replica: d60c8c49796942ca8c9d371629e4ecfe, State: Running, Role: LEADER
I20260924 07:20:00.443372 15790 consensus_queue.cc:237] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55aaec9ea3494fe3818d588d845b4837" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45823 } } peers { permanent_uuid: "d60c8c49796942ca8c9d371629e4ecfe" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 44921 } } peers { permanent_uuid: "672eb8a049864e7cb2df4370d1db33cf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42989 } }
I20260924 07:20:00.444201 15282 catalog_manager.cc:5495] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe reported cstate change: term changed from 0 to 1, leader changed from <none> to d60c8c49796942ca8c9d371629e4ecfe (127.3.246.66). New cstate: current_term: 1 leader_uuid: "d60c8c49796942ca8c9d371629e4ecfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55aaec9ea3494fe3818d588d845b4837" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d60c8c49796942ca8c9d371629e4ecfe" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 44921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672eb8a049864e7cb2df4370d1db33cf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20260924 07:20:00.456670 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.453761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56454 (local address 127.3.246.66:44921)
0924 07:20:00.453807 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:00.453809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:00.453853 (+ 44us) server_negotiation.cc:407] Connection header received
0924 07:20:00.453965 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:00.453966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:00.453990 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:00.454055 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:00.454236 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.454724 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.455624 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:00.455892 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:00.455962 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:00.456178 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:00.456320 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:00.456399 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:00.456565 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 07:20:00.456588 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:20:00.466437 15449 raft_consensus.cc:1261] T 66cbe6271ea94a31978183b8d3d8860e P 55aaec9ea3494fe3818d588d845b4837 [term 1 FOLLOWER]: Refusing update from remote peer d60c8c49796942ca8c9d371629e4ecfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:00.466437 15721 raft_consensus.cc:1261] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf [term 1 FOLLOWER]: Refusing update from remote peer d60c8c49796942ca8c9d371629e4ecfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:00.466823 15790 consensus_queue.cc:1035] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "55aaec9ea3494fe3818d588d845b4837" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 45823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:00.466992 15790 consensus_queue.cc:1035] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "672eb8a049864e7cb2df4370d1db33cf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:00.470149 15798 mvcc.cc:205] Tried to move back new op lower bound from 7332800104279236608 to 7332800104216612864. Current Snapshot: MvccSnapshot[applied={T|T < 7332800104279236608}]
I20260924 07:20:00.470671 15800 mvcc.cc:205] Tried to move back new op lower bound from 7332800104279236608 to 7332800104216612864. Current Snapshot: MvccSnapshot[applied={T|T < 7332800104279236608}]
I20260924 07:20:00.471866 15799 mvcc.cc:205] Tried to move back new op lower bound from 7332800104279236608 to 7332800104216612864. Current Snapshot: MvccSnapshot[applied={T|T < 7332800104279236608}]
I20260924 07:20:00.474666 15278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:00.474783 15278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:00.475492 15278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e70a50d8f7b45c0a2283e4bf6c2973f: Sending DeleteTablet for 3 replicas of tablet 66cbe6271ea94a31978183b8d3d8860e
I20260924 07:20:00.475966 15562 tablet_service.cc:1514] Processing DeleteTablet for tablet 66cbe6271ea94a31978183b8d3d8860e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:00 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:56444
I20260924 07:20:00.476032 15429 tablet_service.cc:1514] Processing DeleteTablet for tablet 66cbe6271ea94a31978183b8d3d8860e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:00 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:59264
I20260924 07:20:00.476243 15701 tablet_service.cc:1514] Processing DeleteTablet for tablet 66cbe6271ea94a31978183b8d3d8860e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:00 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:51080
I20260924 07:20:00.476864 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 15364
Sep 24 07:20:00 dist-test-slave-rc1w krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234399, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:00.488134 15810 tablet_replica.cc:330] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf: stopping tablet replica
I20260924 07:20:00.488317 15810 raft_consensus.cc:2229] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:00.488461 15810 raft_consensus.cc:2258] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:00.489107 15810 ts_tablet_manager.cc:1905] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:00.490294 15808 tablet_replica.cc:330] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe: stopping tablet replica
I20260924 07:20:00.490422 15808 raft_consensus.cc:2229] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:20:00.490571 15808 raft_consensus.cc:2258] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 07:20:00.490942 15267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:45823 (error 108)
I20260924 07:20:00.491348 15810 ts_tablet_manager.cc:1918] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:00.491484 15808 ts_tablet_manager.cc:1905] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:00.491513 15810 log.cc:1196] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/66cbe6271ea94a31978183b8d3d8860e
I20260924 07:20:00.492242 15810 ts_tablet_manager.cc:1939] T 66cbe6271ea94a31978183b8d3d8860e P 672eb8a049864e7cb2df4370d1db33cf: Deleting consensus metadata
I20260924 07:20:00.492765 15266 catalog_manager.cc:4841] TS 672eb8a049864e7cb2df4370d1db33cf (127.3.246.67:42989): tablet 66cbe6271ea94a31978183b8d3d8860e (table test-table [id=f35fd0efdab3401cb2b8e631b41f0f96]) successfully deleted
I20260924 07:20:00.493563 15808 ts_tablet_manager.cc:1918] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:00.493737 15808 log.cc:1196] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/66cbe6271ea94a31978183b8d3d8860e
I20260924 07:20:00.494130 15808 ts_tablet_manager.cc:1939] T 66cbe6271ea94a31978183b8d3d8860e P d60c8c49796942ca8c9d371629e4ecfe: Deleting consensus metadata
I20260924 07:20:00.494673 15267 catalog_manager.cc:4841] TS d60c8c49796942ca8c9d371629e4ecfe (127.3.246.66:44921): tablet 66cbe6271ea94a31978183b8d3d8860e (table test-table [id=f35fd0efdab3401cb2b8e631b41f0f96]) successfully deleted
I20260924 07:20:00.501982 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 15503
I20260924 07:20:00.502153 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.502006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:45823 (local address 127.0.0.1:59280)
0924 07:20:00.502069 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:00.502110 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:45823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 07:20:00.505088 15267 catalog_manager.cc:4584] TS 55aaec9ea3494fe3818d588d845b4837 (127.3.246.65:45823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66cbe6271ea94a31978183b8d3d8860e: Network error: Client connection negotiation failed: client connection to 127.3.246.65:45823: connect: Connection refused (error 111)
I20260924 07:20:00.522320 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 15638
I20260924 07:20:00.528208 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 15250
I20260924 07:20:00.535009 4057 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 07:20:00.879657 4057 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 07:20:00.879760 4057 mini_ranger.cc:66] Stopping Ranger...
I20260924 07:20:01.224826 4057 mini_ranger.cc:68] Stopped Ranger
2026-09-24 07:20:01.225 UTC [14836] LOG: received smart shutdown request
2026-09-24 07:20:01.228 UTC [14836] LOG: background worker "logical replication launcher" (PID 14845) exited with exit code 1
2026-09-24 07:20:01.229 UTC [14840] LOG: shutting down
2026-09-24 07:20:01.414 UTC [14836] LOG: database system is shut down
2026-09-24T07:20:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44735 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:01 dist-test-slave-rc1w krb5kdc[15820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:01 dist-test-slave-rc1w krb5kdc[15820](info): set up 2 sockets
Sep 24 07:20:01 dist-test-slave-rc1w krb5kdc[15820](info): commencing operation
krb5kdc: starting...
W20260924 07:20:02.561833 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:20:02 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/rangeradmin_127.3.246.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/rangeradmin_127.3.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/rangerlookup_127.3.246.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/rangerlookup_127.3.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/HTTP_127.3.246.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/HTTP_127.3.246.84@KRBTEST.COM.keytab.
I20260924 07:20:02.728822 4057 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwu82JT/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/postgres -l logfile start
2026-09-24 07:20:05.321 UTC [15850] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 07:20:05.321 UTC [15850] LOG: listening on IPv4 address "127.3.246.84", port 44159
2026-09-24 07:20:05.325 UTC [15850] LOG: listening on Unix socket "/tmp/.s.PGSQL.44159"
2026-09-24 07:20:05.360 UTC [15853] LOG: database system was shut down at 2026-09-24 07:20:03 UTC
2026-09-24 07:20:05.364 UTC [15850] LOG: database system is ready to accept connections
I20260924 07:20:06.301769 4057 mini_postgres.cc:96] Postgres bound to 44159
2026-09-24 07:20:06.307 UTC [15861] FATAL: database "slave" does not exist
127.3.246.84:44159 - accepting connections
I20260924 07:20:06.308362 4057 mini_ranger.cc:161] Starting Ranger...
I20260924 07:20:06.323554 4057 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 07:20:06.839236 4057 mini_ranger.cc:87] Created ranger Postgres database
I20260924 07:20:06.839350 4057 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-admin
2026-09-24 07:20:07,112 [I] DB FLAVOR :POSTGRES
2026-09-24 07:20:07,112 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:20:07,113 [I] Checking connection..
2026-09-24 07:20:07,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 07:20:07,359 [I] Checking connection passed.
2026-09-24 07:20:07,359 [I] --------- Verifying version history table ---------
2026-09-24 07:20:07,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:20:07,587 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:20:07,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:20:07,820 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:20:07,821 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 07:20:07,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 07:20:08.059 UTC [15925] WARNING: there is no transaction in progress
2026-09-24 07:20:08,065 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 07:20:08,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:20:08,296 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 07:20:08,296 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:20:08,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 07:20:08,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 07:20:08,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3m09.c.gcp-upstream.internal','N') ;"
2026-09-24 07:20:08,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:20:09,240 [I] Table x_portal_user does not exist in database ranger
2026-09-24 07:20:09,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:20:09,466 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 07:20:09,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:20:09,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:20:09,928 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 07:20:09,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 07:20:10.889 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:10.906 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:10.922 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:10.937 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:10.955 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:10.975 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:11.038 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:11.044 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:11.059 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:11.069 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:11.079 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:11.093 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:11.104 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:11.111 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:11.119 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:11.126 UTC [16033] WARNING: there is no transaction in progress
2026-09-24 07:20:11,766 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 07:20:11,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 07:20:12,011 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 07:20:12,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:20:12,270 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 07:20:12,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:20:12,512 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 07:20:12,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:20:12,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:20:13,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3m09.c.gcp-upstream.internal';"
2026-09-24 07:20:13,255 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 07:20:13,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 07:20:13,506 [I] DB_PATCHES have already been applied
I20260924 07:20:13.511765 4057 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwu82JT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 07:20:13.511842 4057 mini_ranger.cc:193] Using host: 127.3.246.84
I20260924 07:20:13.514859 4057 mini_ranger.cc:227] Ranger admin URL: http://127.3.246.84:41159
Sep 24, 2026 7:20:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:20:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:20:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:20:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:20:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:20:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:20:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 07:20:14.614691 4057 mini_ranger.cc:160] Time spent starting Ranger: real 8.306s user 0.000s sys 0.009s
Sep 24, 2026 7:20:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:20:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 7:20:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:20:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:20:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:20:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:20:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:20:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:20:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:20:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:20:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:20:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:20:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:20:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:20:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:20:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:20:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:20:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 07:20:36.335942 4057 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/rangerkms_127.3.246.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/rangerkms_127.3.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.246.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.246.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.246.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/HTTP_127.3.246.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.246.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/HTTP_127.3.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 07:20:36 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234436, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 07:20:36.424459 4057 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 07:20:36.434513 4057 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 07:20:37.378585 4057 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 07:20:37.378665 4057 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-kms
I20260924 07:20:37.378679 4057 mini_ranger_kms.cc:227] Using postgres at 127.3.246.84:44159
2026-09-24 07:20:37,548 [I] DB FLAVOR :POSTGRES
2026-09-24 07:20:37,548 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:20:37,548 [I] Checking connection
2026-09-24 07:20:37,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 07:20:37,776 [I] connection success
2026-09-24 07:20:37,776 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 07:20:37,776 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 07:20:37,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 07:20:38,001 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 07:20:38,002 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:20:38,002 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 07:20:38,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.246.84:44159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 07:20:38,259 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 07:20:38.723186 4057 mini_ranger_kms.cc:310] Created kms service
I20260924 07:20:38.901559 4057 mini_ranger_kms.cc:326] Created kudu user
I20260924 07:20:38.946781 4057 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 07:20:38 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234436, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:39.343477 4057 mini_ranger_kms.cc:384] Added ranger policy
I20260924 07:20:39.343683 4057 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwu82JT/build/release/bin/postgresql.jar:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwu82JT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 07:20:39.343705 4057 mini_ranger_kms.cc:240] Using host: 127.3.246.84
I20260924 07:20:39.346942 4057 mini_ranger_kms.cc:276] Ranger KMS PID: 16213
I20260924 07:20:39.347021 4057 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.246.84:51361
Sep 24, 2026 7:20:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:20:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:20:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:20:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:20:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:20:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
log4j:WARN getHostname()
java.net.UnknownHostException: dist-test-slave-3m09.c.gcp-upstream.internal: dist-test-slave-3m09.c.gcp-upstream.internal: Name or service not known
at java.net.InetAddress.getLocalHost(InetAddress.java:1432)
at org.apache.ranger.audit.provider.MiscUtil.initLocalHost(MiscUtil.java:843)
at org.apache.ranger.audit.provider.MiscUtil.<clinit>(MiscUtil.java:100)
at org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer.authWithKerberos(RangerKmsAuthorizer.java:150)
at org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer.<init>(RangerKmsAuthorizer.java:126)
at org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer.<init>(RangerKmsAuthorizer.java:154)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer.init(RangerKmsAuthorizer.java:71)
at org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer.<init>(RangerKmsAuthorizer.java:51)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.crypto.key.kms.server.KMSWebApp.getAcls(KMSWebApp.java:240)
at org.apache.hadoop.crypto.key.kms.server.KMSWebApp.contextInitialized(KMSWebApp.java:139)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5128)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5653)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1689)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1679)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.net.UnknownHostException: dist-test-slave-3m09.c.gcp-upstream.internal: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:867)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:815)
at java.net.InetAddress.getAllByName0(InetAddress.java:1291)
at java.net.InetAddress.getLocalHost(InetAddress.java:1427)
... 28 more
Sep 24 07:20:41 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790234441, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.246.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:41.609957 4057 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.185s user 0.012s sys 0.140s
I20260924 07:20:41.610057 4057 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 07:20:41.610109 4057 mini_ranger_kms.cc:401] 127.3.246.84:51361/kms/v1/keys
Sep 24 07:20:42 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234441, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.246.84@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
Sep 24 07:20:42 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234441, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.246.84@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
Sep 24 07:20:44 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:20:44.592962 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:42865
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:42865
--ranger_config_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.246.84:51361/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:44.674046 16264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:44.674288 16264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:44.674328 16264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:44.675496 16264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:44.675570 16264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:44.675599 16264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:44.675657 16264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:44.675683 16264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:44.677244 16264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.246.84:51361/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:42865
--ranger_config_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:42865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:44.677510 16264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:44.677825 16264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:44.683414 16270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:44.683420 16264 server_base.cc:1034] running on GCE node
W20260924 07:20:44.683511 16269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:44.683701 16272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:44.684008 16264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:44.684357 16264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:44.685527 16264 hybrid_clock.cc:648] HybridClock initialized: now 1790234444685507 us; error 44 us; skew 500 ppm
Sep 24 07:20:44 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:44.688218 16264 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:20:44.688745 16264 webserver.cc:460] Webserver started at http://127.3.246.126:41211/ using document root <none> and password file <none>
I20260924 07:20:44.688998 16264 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:44.689061 16264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:44.689214 16264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:20:44 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:44.742762 16264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "d02639aef31e4485afea65985b2e0540"
format_stamp: "Formatted at 2026-09-24 07:20:44 on dist-test-slave-rc1w"
server_key: "88c31f9a961a51f0c26ff25f364b8c58"
server_key_iv: "85274117a2a8b7e848e7d76403b5cf83"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:44.743235 16264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "d02639aef31e4485afea65985b2e0540"
format_stamp: "Formatted at 2026-09-24 07:20:44 on dist-test-slave-rc1w"
server_key: "88c31f9a961a51f0c26ff25f364b8c58"
server_key_iv: "85274117a2a8b7e848e7d76403b5cf83"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:44.745105 16264 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:20:44.759307 16279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:44.759505 16264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:20:44.759596 16264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "d02639aef31e4485afea65985b2e0540"
format_stamp: "Formatted at 2026-09-24 07:20:44 on dist-test-slave-rc1w"
server_key: "88c31f9a961a51f0c26ff25f364b8c58"
server_key_iv: "85274117a2a8b7e848e7d76403b5cf83"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:44.759701 16264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:44.771525 16264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:44.773655 16264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:44.773875 16264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:44.779006 16264 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:42865
I20260924 07:20:44.779085 16331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:42865 every 8 connection(s)
I20260924 07:20:44.779450 16264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:44.780145 16332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:44.782529 16332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540: Bootstrap starting.
I20260924 07:20:44.783114 16332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:44.783416 16332 log.cc:824] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:44.784077 16332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540: No bootstrap required, opened a new log
I20260924 07:20:44.786139 16332 raft_consensus.cc:348] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02639aef31e4485afea65985b2e0540" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42865 } }
I20260924 07:20:44.786283 16332 raft_consensus.cc:374] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:44.786305 16332 raft_consensus.cc:729] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02639aef31e4485afea65985b2e0540, State: Initialized, Role: FOLLOWER
I20260924 07:20:44.786460 16332 consensus_queue.cc:260] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02639aef31e4485afea65985b2e0540" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42865 } }
I20260924 07:20:44.786532 16332 raft_consensus.cc:388] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:44.786554 16332 raft_consensus.cc:482] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:44.786621 16332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:44.787173 16332 raft_consensus.cc:504] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02639aef31e4485afea65985b2e0540" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42865 } }
I20260924 07:20:44.787256 16332 leader_election.cc:302] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d02639aef31e4485afea65985b2e0540; no voters:
I20260924 07:20:44.787479 16332 leader_election.cc:288] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:44.787503 16335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:44.787617 16335 raft_consensus.cc:686] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 1 LEADER]: Becoming Leader. State: Replica: d02639aef31e4485afea65985b2e0540, State: Running, Role: LEADER
I20260924 07:20:44.787693 16335 consensus_queue.cc:237] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02639aef31e4485afea65985b2e0540" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42865 } }
I20260924 07:20:44.787731 16332 sys_catalog.cc:564] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:44.788012 16335 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d02639aef31e4485afea65985b2e0540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02639aef31e4485afea65985b2e0540" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42865 } } }
I20260924 07:20:44.788188 16335 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:44.788365 16335 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d02639aef31e4485afea65985b2e0540. Latest consensus state: current_term: 1 leader_uuid: "d02639aef31e4485afea65985b2e0540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02639aef31e4485afea65985b2e0540" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42865 } } }
I20260924 07:20:44.788486 16335 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:44.788776 16343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:44.788921 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 16264
I20260924 07:20:44.789036 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:20:44.789149 16332 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:20:44.789240 4057 external_mini_cluster.cc:1428] Setting key a2e935b0bc307bdae845d8751c61a672
I20260924 07:20:44.789315 16343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:44.798967 16343 catalog_manager.cc:1347] Generated new cluster ID: 59a2f63233d0471b9c23a92b4bc2f153
I20260924 07:20:44.799055 16343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:20:44 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:44.807828 16343 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:44.808668 16343 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:44.814213 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.791603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.3.246.126:42865)
0924 07:20:44.797830 (+ 6227us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.797840 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.797883 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:20:44.798036 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.798044 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.798106 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.798299 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.799200 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.800679 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.801831 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.802454 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.802549 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.805381 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.805386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.805386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.805398 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.807416 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.807821 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.807824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.807824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.808869 (+ 1045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.809093 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.809094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.809095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.809394 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.809541 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.813795 (+ 4254us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.813965 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6160,"thread_start_us":275,"threads_started":1}
I20260924 07:20:44.816723 16343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540: Generated new TSK 0
I20260924 07:20:44.816946 16343 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:20:45.503898 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.246.84:51361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42865
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:45.591291 16377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:45.591521 16377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:45.591557 16377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:45.592689 16377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:45.592756 16377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:45.592799 16377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:20:45.594247 16377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.246.84:51361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:45.594502 16377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:45.594722 16377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:45.600798 16385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:45.600822 16382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:45.600823 16383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:45.601145 16377 server_base.cc:1034] running on GCE node
I20260924 07:20:45.601377 16377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:45.601709 16377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:45.602866 16377 hybrid_clock.cc:648] HybridClock initialized: now 1790234445602856 us; error 27 us; skew 500 ppm
Sep 24 07:20:45 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:45.605381 16377 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:20:45.605929 16377 webserver.cc:460] Webserver started at http://127.3.246.65:42437/ using document root <none> and password file <none>
I20260924 07:20:45.606177 16377 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:45.606221 16377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:45.606352 16377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:20:45 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:45.620821 16377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "6e7560e2abc34126985999eeb0525c1c"
format_stamp: "Formatted at 2026-09-24 07:20:45 on dist-test-slave-rc1w"
server_key: "2e274b5ae4cbe3214a2bf5d7818d1b82"
server_key_iv: "5065f2e5c2abfbd783068ff5e4f91e31"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:45.621567 16377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "6e7560e2abc34126985999eeb0525c1c"
format_stamp: "Formatted at 2026-09-24 07:20:45 on dist-test-slave-rc1w"
server_key: "2e274b5ae4cbe3214a2bf5d7818d1b82"
server_key_iv: "5065f2e5c2abfbd783068ff5e4f91e31"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:45.627967 16377 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260924 07:20:45.642572 16392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:45.642848 16377 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:45.642971 16377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "6e7560e2abc34126985999eeb0525c1c"
format_stamp: "Formatted at 2026-09-24 07:20:45 on dist-test-slave-rc1w"
server_key: "2e274b5ae4cbe3214a2bf5d7818d1b82"
server_key_iv: "5065f2e5c2abfbd783068ff5e4f91e31"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:45.643138 16377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:45.663481 16377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:45.665622 16377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:45.665817 16377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:45.666081 16377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:45.666438 16377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:45.666493 16377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:45.666536 16377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:45.666563 16377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:45.673082 16377 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:42449
I20260924 07:20:45.673152 16505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:42449 every 8 connection(s)
I20260924 07:20:45.673573 16377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:20:45.676514 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 16377
I20260924 07:20:45.676609 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:20:45.676810 4057 external_mini_cluster.cc:1428] Setting key 040d6170cee1c90b6001dffdaba731a8
Sep 24 07:20:45 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:45.688496 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:45.674325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42865 (local address 127.3.246.65:52227)
0924 07:20:45.674506 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:45.674548 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:45.675157 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:45.675442 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:45.675445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:45.675632 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:45.675843 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:45.675847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.676764 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.676765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:45.677971 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:45.677974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.682236 (+ 4262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.682238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:45.682752 (+ 514us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:45.683255 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:45.683266 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:45.684794 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:45.686847 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:45.686853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:45.686856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:45.687149 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:45.687381 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:45.687382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:45.687382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:45.687511 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:45.687831 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:45.687838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:45.687994 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 07:20:45.688101 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:20:45.688266 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260924 07:20:45.689193 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:45.674638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:52227 (local address 127.3.246.126:42865)
0924 07:20:45.674792 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:45.674794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:45.674805 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:20:45.675284 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:45.675285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:45.675301 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:45.675367 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:45.675939 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.676650 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.678072 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.678342 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.682145 (+ 3803us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:45.684907 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:45.684909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:45.684909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:45.684917 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:45.686749 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:45.687240 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:45.687242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:45.687242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:45.687309 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:45.687588 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:45.687589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:45.687590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:45.687740 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:45.687804 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:45.689060 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0924 07:20:45.689130 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260924 07:20:45.689558 16506 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42865
I20260924 07:20:45.689702 16506 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:45.689920 16506 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
I20260924 07:20:45.690464 16296 ts_manager.cc:194] Registered new tserver with Master: 6e7560e2abc34126985999eeb0525c1c (127.3.246.65:42449)
I20260924 07:20:45.691431 16296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:52227
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:20:45.717355 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.246.84:51361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42865
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:45.820937 16514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:45.821158 16514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:45.821197 16514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:45.822539 16514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:45.822609 16514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:45.822654 16514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:20:45.824069 16514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.246.84:51361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:45.824357 16514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:45.824613 16514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:45.829526 16522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:45.829527 16524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:45.829854 16521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:45.830250 16514 server_base.cc:1034] running on GCE node
I20260924 07:20:45.830492 16514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:45.830768 16514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:45.834022 16514 hybrid_clock.cc:648] HybridClock initialized: now 1790234445834003 us; error 27 us; skew 500 ppm
Sep 24 07:20:45 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:45.836546 16514 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:20:45.836961 16514 webserver.cc:460] Webserver started at http://127.3.246.66:43807/ using document root <none> and password file <none>
I20260924 07:20:45.837205 16514 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:45.837272 16514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:45.837409 16514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:20:45 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:45.852213 16514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a"
format_stamp: "Formatted at 2026-09-24 07:20:45 on dist-test-slave-rc1w"
server_key: "5d9eef7d8a075e88d8225428e350d9c9"
server_key_iv: "ba59549430665a6e4bee5ad3ac41a93a"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:45.852730 16514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a"
format_stamp: "Formatted at 2026-09-24 07:20:45 on dist-test-slave-rc1w"
server_key: "5d9eef7d8a075e88d8225428e350d9c9"
server_key_iv: "ba59549430665a6e4bee5ad3ac41a93a"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:45.854936 16514 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
Sep 24 07:20:45 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790234445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:45.873379 16531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:45.876427 16514 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260924 07:20:45.876544 16514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a"
format_stamp: "Formatted at 2026-09-24 07:20:45 on dist-test-slave-rc1w"
server_key: "5d9eef7d8a075e88d8225428e350d9c9"
server_key_iv: "ba59549430665a6e4bee5ad3ac41a93a"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:45.876655 16514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:45.908435 16514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:45.910964 16514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:45.911181 16514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:45.911500 16514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:45.911900 16514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:45.911983 16514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:45.912050 16514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:45.912102 16514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:45.919581 16514 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:37169
I20260924 07:20:45.920077 16514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:20:45.922051 16644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:37169 every 8 connection(s)
I20260924 07:20:45.925086 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 16514
I20260924 07:20:45.925182 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:20:45.925391 4057 external_mini_cluster.cc:1428] Setting key 77b4c557a02d74a2f2087e02c97af3e3
Sep 24 07:20:45 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
I20260924 07:20:45.943308 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:45.921562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:56527 (local address 127.3.246.126:42865)
0924 07:20:45.921608 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:45.921611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:45.922402 (+ 791us) server_negotiation.cc:407] Connection header received
0924 07:20:45.923104 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:45.923106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:45.923123 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:45.923155 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:45.923994 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.924819 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.929148 (+ 4329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.929475 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.929524 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:45.939896 (+ 10372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:45.939898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:45.939898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:45.939907 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:45.941596 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:45.942135 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:45.942137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:45.942137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:45.942202 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:45.942492 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:45.942493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:45.942493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:45.942647 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:45.942710 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:45.943107 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 07:20:45.943204 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:20:45.943477 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:45.921284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42865 (local address 127.3.246.66:56527)
0924 07:20:45.922258 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:45.922313 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:45.922970 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:45.923359 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:45.923363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:45.923604 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:45.923875 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:45.923881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.927786 (+ 3905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.927786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:45.929044 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:45.929047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.929637 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.929638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:45.929805 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:45.938050 (+ 8245us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:45.938062 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:45.939771 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:45.941715 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:45.941722 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:45.941725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:45.942025 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:45.942278 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:45.942279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:45.942279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:45.942407 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:45.942742 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:45.942751 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:45.942930 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 07:20:45.943057 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 07:20:45.943226 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"thread_start_us":81,"threads_started":1}
I20260924 07:20:45.944134 16645 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42865
I20260924 07:20:45.944243 16645 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:45.944485 16645 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
I20260924 07:20:45.944875 16296 ts_manager.cc:194] Registered new tserver with Master: 23f39637de1c4d3ca3bd8c9af63a5b2a (127.3.246.66:37169)
I20260924 07:20:45.945408 16296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:56527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:20:45.964497 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.246.84:51361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42865
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:46.081734 16651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:46.082098 16651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:46.082191 16651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:46.083916 16651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:46.084124 16651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:46.084211 16651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:20:46.086380 16651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.246.84:51361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:46.086843 16651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:46.087157 16651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:46.093940 16656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:46.093940 16657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:46.094002 16659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:46.095289 16651 server_base.cc:1034] running on GCE node
I20260924 07:20:46.095502 16651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:46.095734 16651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:46.096897 16651 hybrid_clock.cc:648] HybridClock initialized: now 1790234446096872 us; error 37 us; skew 500 ppm
Sep 24 07:20:46 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:46.099766 16651 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:20:46.100286 16651 webserver.cc:460] Webserver started at http://127.3.246.67:41919/ using document root <none> and password file <none>
I20260924 07:20:46.100540 16651 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:46.100589 16651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:46.100711 16651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:20:46 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:46.114476 16651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "fec832720c994af4bc0923137b24e2d6"
format_stamp: "Formatted at 2026-09-24 07:20:46 on dist-test-slave-rc1w"
server_key: "c5a24782f8539b3e30920ee5c010af14"
server_key_iv: "349bb0dcddb4b954de76f5bb641dbad7"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:46.115005 16651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "fec832720c994af4bc0923137b24e2d6"
format_stamp: "Formatted at 2026-09-24 07:20:46 on dist-test-slave-rc1w"
server_key: "c5a24782f8539b3e30920ee5c010af14"
server_key_iv: "349bb0dcddb4b954de76f5bb641dbad7"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:46.117532 16651 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:20:46.134869 16666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:46.139000 16651 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260924 07:20:46.139133 16651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "fec832720c994af4bc0923137b24e2d6"
format_stamp: "Formatted at 2026-09-24 07:20:46 on dist-test-slave-rc1w"
server_key: "c5a24782f8539b3e30920ee5c010af14"
server_key_iv: "349bb0dcddb4b954de76f5bb641dbad7"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:46.139238 16651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:46.157100 16651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:46.159713 16651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:46.159938 16651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:46.160235 16651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:46.160674 16651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:46.160768 16651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:46.160869 16651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:46.160934 16651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:46.167603 16651 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:36127
I20260924 07:20:46.168085 16651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:20:46.172348 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 16651
I20260924 07:20:46.172465 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:20:46.172698 4057 external_mini_cluster.cc:1428] Setting key ef886da8d279b1141ab824cfea3a853e
I20260924 07:20:46.174609 16779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:36127 every 8 connection(s)
Sep 24 07:20:46 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:46.184275 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:46.175298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42865 (local address 127.3.246.67:41503)
0924 07:20:46.175498 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:46.175542 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:46.176046 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:46.176284 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:46.176287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:46.176481 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:46.176672 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:46.176675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:46.177377 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:46.177378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:46.178277 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:46.178279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:46.178687 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:46.178688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:46.178854 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:46.179425 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:46.179437 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:46.181005 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:46.182775 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:46.182781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:46.182783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:46.183017 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:46.183241 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:46.183242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:46.183242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:46.183358 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:46.183608 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:46.183614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:46.183756 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 07:20:46.183869 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 07:20:46.184032 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":117,"threads_started":1}
I20260924 07:20:46.184857 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:46.175305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:41503 (local address 127.3.246.126:42865)
0924 07:20:46.175579 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:46.175581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:46.175615 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:20:46.176154 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:46.176156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:46.176167 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:46.176213 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:46.176771 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:46.177281 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:46.178382 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:46.178584 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:46.178666 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:46.181099 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:46.181101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:46.181101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:46.181110 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:46.182644 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:46.183111 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:46.183113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:46.183113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:46.183169 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:46.183427 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:46.183428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:46.183428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:46.183542 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:46.183618 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:46.184719 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0924 07:20:46.184787 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260924 07:20:46.185153 16780 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42865
I20260924 07:20:46.185261 16780 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:46.185436 16780 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
I20260924 07:20:46.185812 16296 ts_manager.cc:194] Registered new tserver with Master: fec832720c994af4bc0923137b24e2d6 (127.3.246.67:36127)
I20260924 07:20:46.186192 16296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:41503
I20260924 07:20:46.195380 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:20:46.195959 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 16377
I20260924 07:20:46.201395 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 16514
I20260924 07:20:46.223518 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 16651
I20260924 07:20:46.230103 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 16264
I20260924 07:20:46.237411 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:42865
--webserver_interface=127.3.246.126
--webserver_port=41211
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:42865
--ranger_config_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.246.84:51361/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.246.84:51361/kms with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:46.317653 16783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:46.317870 16783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:46.317891 16783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:46.319052 16783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:46.319092 16783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:46.319106 16783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:46.319118 16783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:46.319130 16783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:46.320586 16783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.246.84:51361/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:42865
--ranger_config_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:42865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=41211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:46.320758 16783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:46.320912 16783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:46.325568 16791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:46.325557 16789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:46.325558 16788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:46.325944 16783 server_base.cc:1034] running on GCE node
I20260924 07:20:46.326161 16783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:46.326402 16783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:46.327571 16783 hybrid_clock.cc:648] HybridClock initialized: now 1790234446327561 us; error 32 us; skew 500 ppm
Sep 24 07:20:46 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:46.329996 16783 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:20:46.330463 16783 webserver.cc:460] Webserver started at http://127.3.246.126:41211/ using document root <none> and password file <none>
I20260924 07:20:46.330675 16783 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:46.330749 16783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:20:46 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:46.374940 16783 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:46.375803 16799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:46.375994 16783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:46.376096 16783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "d02639aef31e4485afea65985b2e0540"
format_stamp: "Formatted at 2026-09-24 07:20:44 on dist-test-slave-rc1w"
server_key: "88c31f9a961a51f0c26ff25f364b8c58"
server_key_iv: "85274117a2a8b7e848e7d76403b5cf83"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:46.376567 16783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:46.400207 16783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:46.402428 16783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:46.402588 16783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:46.407770 16783 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:42865
I20260924 07:20:46.407841 16851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:42865 every 8 connection(s)
I20260924 07:20:46.408260 16783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:46.409557 16852 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:20:46.410279 16852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540: Bootstrap starting.
I20260924 07:20:46.413590 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 16783
I20260924 07:20:46.414129 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.246.84:51361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:42449
--local_ip_for_outbound_sockets=127.3.246.65
--tserver_master_addrs=127.3.246.126:42865
--webserver_port=42437
--webserver_interface=127.3.246.65
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.246.84:51361/kms with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:20:46.419050 16852 log.cc:824] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:46.420832 16852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:20:46.421142 16852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540: Bootstrap complete.
I20260924 07:20:46.423571 16852 raft_consensus.cc:348] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02639aef31e4485afea65985b2e0540" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42865 } }
I20260924 07:20:46.423831 16852 raft_consensus.cc:729] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02639aef31e4485afea65985b2e0540, State: Initialized, Role: FOLLOWER
I20260924 07:20:46.424039 16852 consensus_queue.cc:260] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02639aef31e4485afea65985b2e0540" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42865 } }
I20260924 07:20:46.424129 16852 raft_consensus.cc:388] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:46.424176 16852 raft_consensus.cc:482] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:46.424216 16852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:20:46.425207 16852 raft_consensus.cc:504] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02639aef31e4485afea65985b2e0540" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42865 } }
I20260924 07:20:46.425333 16852 leader_election.cc:302] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d02639aef31e4485afea65985b2e0540; no voters:
I20260924 07:20:46.425522 16852 leader_election.cc:288] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:20:46.425555 16856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:20:46.425796 16856 raft_consensus.cc:686] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [term 2 LEADER]: Becoming Leader. State: Replica: d02639aef31e4485afea65985b2e0540, State: Running, Role: LEADER
I20260924 07:20:46.425868 16852 sys_catalog.cc:564] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:46.425976 16856 consensus_queue.cc:237] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02639aef31e4485afea65985b2e0540" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42865 } }
I20260924 07:20:46.426311 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d02639aef31e4485afea65985b2e0540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02639aef31e4485afea65985b2e0540" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42865 } } }
I20260924 07:20:46.426469 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:46.426640 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d02639aef31e4485afea65985b2e0540. Latest consensus state: current_term: 2 leader_uuid: "d02639aef31e4485afea65985b2e0540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02639aef31e4485afea65985b2e0540" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42865 } } }
I20260924 07:20:46.426745 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:46.427282 16852 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:20:46.427603 16865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:46.430294 16865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:46.430608 16865 catalog_manager.cc:1259] Loaded cluster ID: 59a2f63233d0471b9c23a92b4bc2f153
I20260924 07:20:46.430698 16865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:46.432756 16865 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:46.433137 16865 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540: Loaded TSK: 0
I20260924 07:20:46.433459 16865 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:20:46.511089 16854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:46.511341 16854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:46.511370 16854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:46.512530 16854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:46.512573 16854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:46.512598 16854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:20:46.514298 16854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.246.84:51361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:42449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=42437
--tserver_master_addrs=127.3.246.126:42865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:46.514559 16854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:46.514798 16854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:46.520188 16887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:46.520262 16888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:46.520785 16890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:46.521286 16854 server_base.cc:1034] running on GCE node
I20260924 07:20:46.521570 16854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:46.521852 16854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:46.525120 16854 hybrid_clock.cc:648] HybridClock initialized: now 1790234446525099 us; error 38 us; skew 500 ppm
Sep 24 07:20:46 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:46.528224 16854 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:20:46.528687 16854 webserver.cc:460] Webserver started at http://127.3.246.65:42437/ using document root <none> and password file <none>
I20260924 07:20:46.528941 16854 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:46.529026 16854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:20:46 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:46.547429 16854 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:46.550145 16900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:46.550371 16854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:20:46.550470 16854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "6e7560e2abc34126985999eeb0525c1c"
format_stamp: "Formatted at 2026-09-24 07:20:45 on dist-test-slave-rc1w"
server_key: "2e274b5ae4cbe3214a2bf5d7818d1b82"
server_key_iv: "5065f2e5c2abfbd783068ff5e4f91e31"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:46.550931 16854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:46.566156 16854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:46.569046 16854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:46.569212 16854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:46.569489 16854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:46.572613 16854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:46.572897 16854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:46.572952 16854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:46.572978 16854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:46.579551 16854 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:42449
I20260924 07:20:46.580013 16854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:20:46.583217 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 16854
I20260924 07:20:46.583532 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.246.84:51361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:37169
--local_ip_for_outbound_sockets=127.3.246.66
--tserver_master_addrs=127.3.246.126:42865
--webserver_port=43807
--webserver_interface=127.3.246.66
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.246.84:51361/kms with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:20:46.587244 17013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:42449 every 8 connection(s)
Sep 24 07:20:46 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:46.604198 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:46.587920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42865 (local address 127.3.246.65:52081)
0924 07:20:46.588435 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:46.588478 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:46.589101 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:46.589858 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:46.589863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:46.590102 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:46.590383 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:46.590401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:46.592500 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:46.592501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:46.593817 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:46.593820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:46.594729 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:46.594730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:46.594862 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:46.595524 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:46.595533 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:46.597210 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:46.599772 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:46.599779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:46.599781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:46.600872 (+ 1091us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:46.601401 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:46.601402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:46.601402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:46.601539 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:46.603147 (+ 1608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:46.603515 (+ 368us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:46.603678 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 07:20:46.603791 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 07:20:46.603960 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":87,"threads_started":1}
I20260924 07:20:46.605190 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:46.588030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:52081 (local address 127.3.246.126:42865)
0924 07:20:46.588334 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:46.588344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:46.589267 (+ 923us) server_negotiation.cc:407] Connection header received
0924 07:20:46.589452 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:46.589460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:46.589519 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:46.589653 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:46.590525 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:46.592361 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:46.593915 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:46.594523 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:46.594621 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:46.597354 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:46.597358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:46.597359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:46.597370 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:46.599660 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:46.601187 (+ 1527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:46.601190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:46.601191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:46.601315 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:46.601615 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:46.601616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:46.601617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:46.602624 (+ 1007us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:46.604593 (+ 1969us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:46.604827 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 07:20:46.604958 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":96,"threads_started":1}
I20260924 07:20:46.610078 17014 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42865
I20260924 07:20:46.610196 17014 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:46.610399 17014 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
W20260924 07:20:46.611363 17014 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:42865 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:20:46.611431 17014 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:46.611490 17014 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
I20260924 07:20:46.611858 17014 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:46.611924 17014 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
W20260924 07:20:46.612226 17014 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:20:46.704044 17016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:46.704442 17016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:46.704538 17016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:46.706354 17016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:46.706423 17016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:46.706465 17016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:20:46.708770 17016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.246.84:51361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:37169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=43807
--tserver_master_addrs=127.3.246.126:42865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:46.709069 17016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:46.709393 17016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:46.714854 17026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:46.714874 17024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:46.715449 17023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:46.715983 17016 server_base.cc:1034] running on GCE node
I20260924 07:20:46.716205 17016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:46.716485 17016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:46.717655 17016 hybrid_clock.cc:648] HybridClock initialized: now 1790234446717626 us; error 43 us; skew 500 ppm
Sep 24 07:20:46 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:46.720819 17016 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:20:46.721360 17016 webserver.cc:460] Webserver started at http://127.3.246.66:43807/ using document root <none> and password file <none>
I20260924 07:20:46.721700 17016 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:46.721791 17016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:20:46 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:46.750291 17016 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260924 07:20:46.754806 17034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:46.755030 17016 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 07:20:46.755131 17016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a"
format_stamp: "Formatted at 2026-09-24 07:20:45 on dist-test-slave-rc1w"
server_key: "5d9eef7d8a075e88d8225428e350d9c9"
server_key_iv: "ba59549430665a6e4bee5ad3ac41a93a"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:46.755625 17016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:46.788354 17016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:46.790653 17016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:46.790823 17016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:46.791095 17016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:46.791483 17016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:46.791555 17016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:46.791595 17016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:46.791622 17016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:46.798790 17016 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:37169
I20260924 07:20:46.799163 17016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:20:46.799768 17147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:37169 every 8 connection(s)
I20260924 07:20:46.801853 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 17016
I20260924 07:20:46.802124 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.246.84:51361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:36127
--local_ip_for_outbound_sockets=127.3.246.67
--tserver_master_addrs=127.3.246.126:42865
--webserver_port=41919
--webserver_interface=127.3.246.67
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.246.84:51361/kms with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:20:46 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:46.810737 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:46.799900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:35609 (local address 127.3.246.126:42865)
0924 07:20:46.799949 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:46.799951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:46.800202 (+ 251us) server_negotiation.cc:407] Connection header received
0924 07:20:46.800757 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:46.800758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:46.800774 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:46.800838 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:46.801390 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:46.802244 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:46.804958 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:46.805191 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:46.805313 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:46.807644 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:46.807645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:46.807646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:46.807653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:46.809162 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:46.809593 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:46.809595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:46.809595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:46.809656 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:46.809961 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:46.809962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:46.809962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:46.810100 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:46.810166 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:46.810527 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 07:20:46.810644 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:20:46.811626 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:46.799916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42865 (local address 127.3.246.66:35609)
0924 07:20:46.800074 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:46.800114 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:46.800645 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:46.800910 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:46.800913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:46.801096 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:46.801289 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:46.801293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:46.803636 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:46.803638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:46.804851 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:46.804854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:46.805343 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:46.805344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:46.805503 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:46.806079 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:46.806087 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:46.807530 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:46.809268 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:46.809276 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:46.809278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:46.809512 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:46.809770 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:46.809770 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:46.809771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:46.809868 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:46.810195 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:46.810202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:46.810365 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 07:20:46.811224 (+ 859us) client_negotiation.cc:240] Negotiation successful
0924 07:20:46.811391 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260924 07:20:46.812156 17148 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42865
I20260924 07:20:46.812273 17148 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:46.812456 17148 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
W20260924 07:20:46.813110 17148 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:42865 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:20:46.813194 17148 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:46.813265 17148 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
I20260924 07:20:46.813587 17148 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:46.813663 17148 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
W20260924 07:20:46.813988 17148 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:20:46.924077 17151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:46.924342 17151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:46.924381 17151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:46.926116 17151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:46.926170 17151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:46.926213 17151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:20:46.928283 17151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.246.84:51361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:36127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=41919
--tserver_master_addrs=127.3.246.126:42865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:46.928507 17151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:46.928755 17151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:46.935024 17157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:46.935060 17156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:46.935626 17151 server_base.cc:1034] running on GCE node
W20260924 07:20:46.937815 17159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:46.938061 17151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:46.938331 17151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:46.941756 17151 hybrid_clock.cc:648] HybridClock initialized: now 1790234446941735 us; error 33 us; skew 500 ppm
Sep 24 07:20:46 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:46.945202 17151 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:20:46.945747 17151 webserver.cc:460] Webserver started at http://127.3.246.67:41919/ using document root <none> and password file <none>
I20260924 07:20:46.946038 17151 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:46.946110 17151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:20:46 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:46.976123 17151 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:46.977053 17168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:46.977229 17151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:46.977352 17151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "fec832720c994af4bc0923137b24e2d6"
format_stamp: "Formatted at 2026-09-24 07:20:46 on dist-test-slave-rc1w"
server_key: "c5a24782f8539b3e30920ee5c010af14"
server_key_iv: "349bb0dcddb4b954de76f5bb641dbad7"
server_key_version: "kuduclusterkey@0"
I20260924 07:20:46.977774 17151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:46.990406 17151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:46.992487 17151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:46.992633 17151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:46.992861 17151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:46.993185 17151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:46.993220 17151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:46.993248 17151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:46.993266 17151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:47.000240 17151 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:36127
I20260924 07:20:47.000715 17151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:20:47.001272 17281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:36127 every 8 connection(s)
I20260924 07:20:47.006373 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 17151
Sep 24 07:20:47 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:47.012532 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.001578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:58947 (local address 127.3.246.126:42865)
0924 07:20:47.001632 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.001634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.001907 (+ 273us) server_negotiation.cc:407] Connection header received
0924 07:20:47.002614 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.002616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.002630 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.002664 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.003341 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.003854 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.004854 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.005066 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.005155 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.007764 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.007765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.007766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.007774 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.009633 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.010156 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.010157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.010158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.010221 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.011469 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.011470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.011470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.011608 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.011674 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.012346 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.012433 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:20:47.013233 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.001572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42865 (local address 127.3.246.67:58947)
0924 07:20:47.001772 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.001815 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:47.002479 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:47.002854 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:47.002858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:47.003042 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:47.003225 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.003229 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.003955 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.003957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.004753 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.004755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.005184 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.005185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.005303 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.005861 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:47.005869 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:47.007654 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:47.009742 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.009749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.009751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.010055 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.010306 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.010307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.010307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.011403 (+ 1096us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.011705 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:47.011712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:47.012160 (+ 448us) client_negotiation.cc:769] Sending connection context
0924 07:20:47.012840 (+ 680us) client_negotiation.cc:240] Negotiation successful
0924 07:20:47.013004 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":144,"threads_started":1}
I20260924 07:20:47.014022 17282 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42865
I20260924 07:20:47.014164 17282 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:47.014403 17282 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
W20260924 07:20:47.014983 17282 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:42865 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:20:47.015098 17282 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:47.015196 17282 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
I20260924 07:20:47.015532 17282 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:47.015645 17282 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
W20260924 07:20:47.015939 17282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:20:47.016600 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.007128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.3.246.126:42865)
0924 07:20:47.007239 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.007242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.007249 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:20:47.007269 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.007270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.007282 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.007353 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.008299 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.010278 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.011496 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.011821 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.011875 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.013280 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.013281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.013281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.013288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.015102 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.015500 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.015501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.015501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.015555 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.015759 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.015759 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.015760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.015883 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.016381 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.016426 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.016505 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
Sep 24 07:20:47 dist-test-slave-rc1w krb5kdc[15820](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:47.612418 17014 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:47.612582 17014 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
I20260924 07:20:47.612998 16816 ts_manager.cc:194] Registered new tserver with Master: 6e7560e2abc34126985999eeb0525c1c (127.3.246.65:42449)
I20260924 07:20:47.614054 16816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:52081
I20260924 07:20:47.814170 17148 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:47.814334 17148 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
I20260924 07:20:47.814744 16816 ts_manager.cc:194] Registered new tserver with Master: 23f39637de1c4d3ca3bd8c9af63a5b2a (127.3.246.66:37169)
I20260924 07:20:47.815328 16816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:35609
Sep 24 07:20:47 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:48.016162 17282 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:48.016322 17282 heartbeater.cc:511] Master 127.3.246.126:42865 requested a full tablet report, sending...
I20260924 07:20:48.016575 16816 ts_manager.cc:194] Registered new tserver with Master: fec832720c994af4bc0923137b24e2d6 (127.3.246.67:36127)
I20260924 07:20:48.017086 16816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:58947
I20260924 07:20:48.026028 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:20:48.035069 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.027094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54670 (local address 127.3.246.126:42865)
0924 07:20:48.027233 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.027235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.027308 (+ 73us) server_negotiation.cc:407] Connection header received
0924 07:20:48.027368 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.027369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.027383 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.027468 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:48.028194 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.028989 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.030054 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.030374 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.030453 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.031972 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:48.031974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:48.031974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:48.031981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:48.033815 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:48.034209 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:48.034210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:48.034211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:48.034277 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:48.034515 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:48.034516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:48.034516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:48.034655 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:48.034761 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:48.034880 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 07:20:48.034964 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260924 07:20:48.037256 16816 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:20:48.039599 16816 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:20:48.050123 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.042590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:37169 (local address 127.0.0.1:44202)
0924 07:20:48.042817 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:48.042858 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:48.042935 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:48.046016 (+ 3081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:48.046019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:48.046022 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:48.046094 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:48.046096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.047462 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.047463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:48.048809 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:48.048812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.049842 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.049843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:48.049927 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.049941 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 07:20:48.049989 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 07:20:48.050036 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":41,"thread_start_us":76,"threads_started":1}
I20260924 07:20:48.050545 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.042920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.3.246.66:37169)
0924 07:20:48.045819 (+ 2899us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.045824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.045835 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:20:48.045873 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.045875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.045891 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.046164 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:48.046230 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.047355 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.048912 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.049738 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.050333 (+ 595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.050402 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:48.050428 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 07:20:48.050467 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2866,"thread_start_us":67,"threads_started":1}
I20260924 07:20:48.051240 17082 tablet_service.cc:1467] Processing CreateTablet for tablet 80864e4176a04b6191f4ee7eb15e407d (DEFAULT_TABLE table=test-table [id=f12f568d6bb842d7b02a57e0408403a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:48.051625 17082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80864e4176a04b6191f4ee7eb15e407d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:48.052954 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.042810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:42449 (local address 127.0.0.1:35536)
0924 07:20:48.047935 (+ 5125us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:48.047945 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:48.048032 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:48.048288 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:48.048289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:48.048291 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:48.048327 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:48.048328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.049596 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.049597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:48.051652 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:48.051655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.052715 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.052715 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:48.052776 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.052780 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:20:48.052828 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 07:20:48.052865 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5109,"thread_start_us":69,"threads_started":1}
I20260924 07:20:48.052970 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.043078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.3.246.67:36127)
0924 07:20:48.043293 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.043309 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.045859 (+ 2550us) server_negotiation.cc:407] Connection header received
0924 07:20:48.045979 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.045981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.045998 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.046052 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:48.046279 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.047344 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.051727 (+ 4383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.052503 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.052559 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.052616 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:48.052851 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 07:20:48.052892 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":57,"threads_started":1}
I20260924 07:20:48.053406 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.043004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.3.246.65:42449)
0924 07:20:48.043160 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.043166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.048004 (+ 4838us) server_negotiation.cc:407] Connection header received
0924 07:20:48.048116 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.048119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.048135 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.048186 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:48.048401 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.049492 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.051819 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.052551 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.052602 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.052653 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:48.053286 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 07:20:48.053330 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260924 07:20:48.053153 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.042662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:36127 (local address 127.0.0.1:57256)
0924 07:20:48.045783 (+ 3121us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:48.045796 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:48.045890 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:48.046156 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:48.046156 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:48.046158 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:48.046201 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:48.046203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.047447 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.047448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:48.051618 (+ 4170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:48.051621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.052681 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.052682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:48.052746 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.052749 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:20:48.053044 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 07:20:48.053089 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3103,"thread_start_us":98,"threads_started":1}
I20260924 07:20:48.054489 16948 tablet_service.cc:1467] Processing CreateTablet for tablet 80864e4176a04b6191f4ee7eb15e407d (DEFAULT_TABLE table=test-table [id=f12f568d6bb842d7b02a57e0408403a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:48.054863 16948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80864e4176a04b6191f4ee7eb15e407d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:48.055367 17315 tablet_bootstrap.cc:492] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a: Bootstrap starting.
I20260924 07:20:48.056098 17315 tablet_bootstrap.cc:654] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:48.056242 17210 tablet_service.cc:1467] Processing CreateTablet for tablet 80864e4176a04b6191f4ee7eb15e407d (DEFAULT_TABLE table=test-table [id=f12f568d6bb842d7b02a57e0408403a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:48.056586 17210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80864e4176a04b6191f4ee7eb15e407d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:48.056622 17315 log.cc:824] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:48.058705 17317 tablet_bootstrap.cc:492] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c: Bootstrap starting.
I20260924 07:20:48.059345 17317 tablet_bootstrap.cc:654] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:48.059518 17318 tablet_bootstrap.cc:492] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6: Bootstrap starting.
I20260924 07:20:48.059669 17317 log.cc:824] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:48.060243 17318 tablet_bootstrap.cc:654] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:48.060588 17318 log.cc:824] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:48.061319 17318 tablet_bootstrap.cc:492] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6: No bootstrap required, opened a new log
I20260924 07:20:48.061380 17318 ts_tablet_manager.cc:1397] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:20:48.063374 17318 raft_consensus.cc:348] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37169 } } peers { permanent_uuid: "fec832720c994af4bc0923137b24e2d6" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36127 } } peers { permanent_uuid: "6e7560e2abc34126985999eeb0525c1c" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42449 } }
I20260924 07:20:48.063545 17318 raft_consensus.cc:374] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:48.063577 17318 raft_consensus.cc:729] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec832720c994af4bc0923137b24e2d6, State: Initialized, Role: FOLLOWER
I20260924 07:20:48.063685 17318 consensus_queue.cc:260] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37169 } } peers { permanent_uuid: "fec832720c994af4bc0923137b24e2d6" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36127 } } peers { permanent_uuid: "6e7560e2abc34126985999eeb0525c1c" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42449 } }
I20260924 07:20:48.063927 17318 ts_tablet_manager.cc:1428] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:20:48.063971 17282 heartbeater.cc:503] Master 127.3.246.126:42865 was elected leader, sending a full tablet report...
I20260924 07:20:48.064348 17315 tablet_bootstrap.cc:492] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a: No bootstrap required, opened a new log
I20260924 07:20:48.064414 17315 ts_tablet_manager.cc:1397] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 07:20:48.066844 17315 raft_consensus.cc:348] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37169 } } peers { permanent_uuid: "fec832720c994af4bc0923137b24e2d6" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36127 } } peers { permanent_uuid: "6e7560e2abc34126985999eeb0525c1c" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42449 } }
I20260924 07:20:48.067006 17315 raft_consensus.cc:374] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:48.067035 17315 raft_consensus.cc:729] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f39637de1c4d3ca3bd8c9af63a5b2a, State: Initialized, Role: FOLLOWER
I20260924 07:20:48.067144 17315 consensus_queue.cc:260] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37169 } } peers { permanent_uuid: "fec832720c994af4bc0923137b24e2d6" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36127 } } peers { permanent_uuid: "6e7560e2abc34126985999eeb0525c1c" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42449 } }
I20260924 07:20:48.067365 17315 ts_tablet_manager.cc:1428] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:20:48.067466 17148 heartbeater.cc:503] Master 127.3.246.126:42865 was elected leader, sending a full tablet report...
I20260924 07:20:48.068550 17317 tablet_bootstrap.cc:492] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c: No bootstrap required, opened a new log
I20260924 07:20:48.068631 17317 ts_tablet_manager.cc:1397] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260924 07:20:48.071314 17317 raft_consensus.cc:348] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37169 } } peers { permanent_uuid: "fec832720c994af4bc0923137b24e2d6" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36127 } } peers { permanent_uuid: "6e7560e2abc34126985999eeb0525c1c" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42449 } }
I20260924 07:20:48.071478 17317 raft_consensus.cc:374] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:48.071512 17317 raft_consensus.cc:729] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7560e2abc34126985999eeb0525c1c, State: Initialized, Role: FOLLOWER
I20260924 07:20:48.071642 17317 consensus_queue.cc:260] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37169 } } peers { permanent_uuid: "fec832720c994af4bc0923137b24e2d6" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36127 } } peers { permanent_uuid: "6e7560e2abc34126985999eeb0525c1c" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42449 } }
I20260924 07:20:48.071915 17317 ts_tablet_manager.cc:1428] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260924 07:20:48.071967 17014 heartbeater.cc:503] Master 127.3.246.126:42865 was elected leader, sending a full tablet report...
W20260924 07:20:48.089190 17015 tablet.cc:2369] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 07:20:48 dist-test-slave-rc1w krb5kdc[15820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for HTTP/127.3.246.84@KRBTEST.COM
I20260924 07:20:48.155650 17323 raft_consensus.cc:482] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:48.155812 17323 raft_consensus.cc:504] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37169 } } peers { permanent_uuid: "fec832720c994af4bc0923137b24e2d6" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36127 } } peers { permanent_uuid: "6e7560e2abc34126985999eeb0525c1c" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42449 } }
I20260924 07:20:48.156240 17323 leader_election.cc:288] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23f39637de1c4d3ca3bd8c9af63a5b2a (127.3.246.66:37169), fec832720c994af4bc0923137b24e2d6 (127.3.246.67:36127)
I20260924 07:20:48.163161 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.156789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:55005 (local address 127.3.246.66:37169)
0924 07:20:48.156833 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.156835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.156883 (+ 48us) server_negotiation.cc:407] Connection header received
0924 07:20:48.156964 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.156965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.156976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.157046 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:48.157210 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.158024 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.161851 (+ 3827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.162456 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.162504 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.162513 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:48.163036 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 07:20:48.163087 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:20:48.165725 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.156396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:37169 (local address 127.3.246.65:55005)
0924 07:20:48.156828 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:48.156839 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:48.156899 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:48.157065 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:48.157066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:48.157068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:48.157149 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:48.157151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.158313 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.158314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:48.161746 (+ 3432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:48.161750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.162852 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.162853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:48.162947 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.162951 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:20:48.165549 (+ 2598us) client_negotiation.cc:240] Negotiation successful
0924 07:20:48.165597 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":89,"threads_started":1}
I20260924 07:20:48.166205 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.156738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:41799 (local address 127.3.246.67:36127)
0924 07:20:48.159993 (+ 3255us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.159995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.160004 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:20:48.160031 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.160032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.160041 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.160096 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:48.163069 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.163829 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.165252 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.165822 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.165873 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.165882 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:48.166097 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 07:20:48.166138 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3226}
I20260924 07:20:48.166448 17102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80864e4176a04b6191f4ee7eb15e407d" candidate_uuid: "6e7560e2abc34126985999eeb0525c1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a" is_pre_election: true
I20260924 07:20:48.166635 17102 raft_consensus.cc:2454] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7560e2abc34126985999eeb0525c1c in term 0.
I20260924 07:20:48.166937 16903 leader_election.cc:302] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23f39637de1c4d3ca3bd8c9af63a5b2a, 6e7560e2abc34126985999eeb0525c1c; no voters:
I20260924 07:20:48.167080 17323 raft_consensus.cc:2781] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:20:48.167133 17323 raft_consensus.cc:482] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:20:48.167164 17323 raft_consensus.cc:3037] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:48.168095 17323 raft_consensus.cc:504] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37169 } } peers { permanent_uuid: "fec832720c994af4bc0923137b24e2d6" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36127 } } peers { permanent_uuid: "6e7560e2abc34126985999eeb0525c1c" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42449 } }
I20260924 07:20:48.168239 17323 leader_election.cc:288] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [CANDIDATE]: Term 1 election: Requested vote from peers 23f39637de1c4d3ca3bd8c9af63a5b2a (127.3.246.66:37169), fec832720c994af4bc0923137b24e2d6 (127.3.246.67:36127)
I20260924 07:20:48.168418 17102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80864e4176a04b6191f4ee7eb15e407d" candidate_uuid: "6e7560e2abc34126985999eeb0525c1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a"
I20260924 07:20:48.168491 17102 raft_consensus.cc:3037] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:48.169343 17102 raft_consensus.cc:2454] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7560e2abc34126985999eeb0525c1c in term 1.
I20260924 07:20:48.170364 16903 leader_election.cc:302] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23f39637de1c4d3ca3bd8c9af63a5b2a, 6e7560e2abc34126985999eeb0525c1c; no voters:
I20260924 07:20:48.170446 17323 raft_consensus.cc:2781] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:48.170557 17323 raft_consensus.cc:686] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [term 1 LEADER]: Becoming Leader. State: Replica: 6e7560e2abc34126985999eeb0525c1c, State: Running, Role: LEADER
I20260924 07:20:48.170639 17323 consensus_queue.cc:237] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37169 } } peers { permanent_uuid: "fec832720c994af4bc0923137b24e2d6" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36127 } } peers { permanent_uuid: "6e7560e2abc34126985999eeb0525c1c" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42449 } }
I20260924 07:20:48.171178 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.156610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:36127 (local address 127.3.246.65:41799)
0924 07:20:48.158434 (+ 1824us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:48.158444 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:48.158490 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:48.162885 (+ 4395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:48.162886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:48.162889 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:48.162963 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:48.162965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.163982 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.163983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:48.165145 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:48.165148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.165943 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.165944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:48.166013 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.166017 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:20:48.171052 (+ 5035us) client_negotiation.cc:240] Negotiation successful
0924 07:20:48.171101 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1803,"thread_start_us":61,"threads_started":1}
I20260924 07:20:48.171577 17236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80864e4176a04b6191f4ee7eb15e407d" candidate_uuid: "6e7560e2abc34126985999eeb0525c1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec832720c994af4bc0923137b24e2d6" is_pre_election: true
I20260924 07:20:48.171731 17236 raft_consensus.cc:2454] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7560e2abc34126985999eeb0525c1c in term 0.
I20260924 07:20:48.171859 17235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80864e4176a04b6191f4ee7eb15e407d" candidate_uuid: "6e7560e2abc34126985999eeb0525c1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec832720c994af4bc0923137b24e2d6"
I20260924 07:20:48.171931 17235 raft_consensus.cc:3037] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:48.172768 17235 raft_consensus.cc:2454] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7560e2abc34126985999eeb0525c1c in term 1.
I20260924 07:20:48.172991 16816 catalog_manager.cc:5495] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7560e2abc34126985999eeb0525c1c (127.3.246.65). New cstate: current_term: 1 leader_uuid: "6e7560e2abc34126985999eeb0525c1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec832720c994af4bc0923137b24e2d6" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e7560e2abc34126985999eeb0525c1c" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42449 } health_report { overall_health: HEALTHY } } }
I20260924 07:20:48.201182 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.197745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.3.246.65:42449)
0924 07:20:48.197815 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.197818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.197827 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:20:48.197868 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.197869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.197890 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.197976 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:48.198165 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.198925 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.200062 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.200376 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.200433 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.200640 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:48.200778 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:48.200847 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:48.201080 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 07:20:48.201101 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:20:48.203783 17102 raft_consensus.cc:1261] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a [term 1 FOLLOWER]: Refusing update from remote peer 6e7560e2abc34126985999eeb0525c1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:48.203939 17235 raft_consensus.cc:1261] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6 [term 1 FOLLOWER]: Refusing update from remote peer 6e7560e2abc34126985999eeb0525c1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:48.204293 17327 consensus_queue.cc:1035] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec832720c994af4bc0923137b24e2d6" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 36127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:48.204442 17323 consensus_queue.cc:1035] T 80864e4176a04b6191f4ee7eb15e407d P 6e7560e2abc34126985999eeb0525c1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "23f39637de1c4d3ca3bd8c9af63a5b2a" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:48.208863 17331 mvcc.cc:205] Tried to move back new op lower bound from 7332800299839352832 to 7332800299707506688. Current Snapshot: MvccSnapshot[applied={T|T < 7332800299839352832}]
I20260924 07:20:48.209472 17333 mvcc.cc:205] Tried to move back new op lower bound from 7332800299839352832 to 7332800299707506688. Current Snapshot: MvccSnapshot[applied={T|T < 7332800299839352832}]
I20260924 07:20:48.211328 17332 mvcc.cc:205] Tried to move back new op lower bound from 7332800299839352832 to 7332800299707506688. Current Snapshot: MvccSnapshot[applied={T|T < 7332800299839352832}]
I20260924 07:20:48.212947 16816 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:48.213049 16816 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:48.213773 16816 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d02639aef31e4485afea65985b2e0540: Sending DeleteTablet for 3 replicas of tablet 80864e4176a04b6191f4ee7eb15e407d
I20260924 07:20:48.214146 17210 tablet_service.cc:1514] Processing DeleteTablet for tablet 80864e4176a04b6191f4ee7eb15e407d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:48 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:57256
I20260924 07:20:48.214377 17082 tablet_service.cc:1514] Processing DeleteTablet for tablet 80864e4176a04b6191f4ee7eb15e407d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:48 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:44202
I20260924 07:20:48.214463 16948 tablet_service.cc:1514] Processing DeleteTablet for tablet 80864e4176a04b6191f4ee7eb15e407d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:48 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:35536
I20260924 07:20:48.215806 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 16854
I20260924 07:20:48.218686 17340 tablet_replica.cc:330] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6: stopping tablet replica
I20260924 07:20:48.221654 17340 raft_consensus.cc:2229] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:48.221891 17340 raft_consensus.cc:2258] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:48.222710 17341 tablet_replica.cc:330] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a: stopping tablet replica
I20260924 07:20:48.222828 17341 raft_consensus.cc:2229] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:48.222930 17341 raft_consensus.cc:2258] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:48.223623 17341 ts_tablet_manager.cc:1905] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:48.223647 17340 ts_tablet_manager.cc:1905] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:48.225525 17341 ts_tablet_manager.cc:1918] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:48.225600 17341 log.cc:1196] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/80864e4176a04b6191f4ee7eb15e407d
I20260924 07:20:48.225914 17341 ts_tablet_manager.cc:1939] T 80864e4176a04b6191f4ee7eb15e407d P 23f39637de1c4d3ca3bd8c9af63a5b2a: Deleting consensus metadata
I20260924 07:20:48.226357 16802 catalog_manager.cc:4841] TS 23f39637de1c4d3ca3bd8c9af63a5b2a (127.3.246.66:37169): tablet 80864e4176a04b6191f4ee7eb15e407d (table test-table [id=f12f568d6bb842d7b02a57e0408403a2]) successfully deleted
W20260924 07:20:48.226516 16802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:42449 (error 108)
I20260924 07:20:48.226802 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 17016
I20260924 07:20:48.230048 17340 ts_tablet_manager.cc:1918] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:48.230146 17340 log.cc:1196] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/80864e4176a04b6191f4ee7eb15e407d
I20260924 07:20:48.230387 17340 ts_tablet_manager.cc:1939] T 80864e4176a04b6191f4ee7eb15e407d P fec832720c994af4bc0923137b24e2d6: Deleting consensus metadata
I20260924 07:20:48.230824 16801 catalog_manager.cc:4841] TS fec832720c994af4bc0923137b24e2d6 (127.3.246.67:36127): tablet 80864e4176a04b6191f4ee7eb15e407d (table test-table [id=f12f568d6bb842d7b02a57e0408403a2]) successfully deleted
I20260924 07:20:48.231088 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.230970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:42449 (local address 127.0.0.1:35558)
0924 07:20:48.231014 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:48.231052 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 07:20:48.231227 16802 catalog_manager.cc:4584] TS 6e7560e2abc34126985999eeb0525c1c (127.3.246.65:42449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80864e4176a04b6191f4ee7eb15e407d: Network error: Client connection negotiation failed: client connection to 127.3.246.65:42449: connect: Connection refused (error 111)
I20260924 07:20:48.231575 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 17151
I20260924 07:20:48.237038 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 16783
I20260924 07:20:48.244493 4057 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 07:20:48.588812 4057 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 07:20:48.588946 4057 mini_ranger.cc:66] Stopping Ranger...
I20260924 07:20:48.933099 4057 mini_ranger.cc:68] Stopped Ranger
2026-09-24 07:20:48.933 UTC [15850] LOG: received smart shutdown request
2026-09-24 07:20:48.936 UTC [15850] LOG: background worker "logical replication launcher" (PID 15859) exited with exit code 1
2026-09-24 07:20:48.937 UTC [15854] LOG: shutting down
2026-09-24 07:20:49.110 UTC [15850] LOG: database system is shut down
2026-09-24T07:20:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47701 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:49 dist-test-slave-rc1w krb5kdc[17352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:49 dist-test-slave-rc1w krb5kdc[17352](info): set up 2 sockets
Sep 24 07:20:49 dist-test-slave-rc1w krb5kdc[17352](info): commencing operation
krb5kdc: starting...
W20260924 07:20:50.257802 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:20:50.372576 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37827
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:37827,127.3.246.125:35619,127.3.246.124:37623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:50.452272 17365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:50.452510 17365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:50.452545 17365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:50.453658 17365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:50.453748 17365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:50.453774 17365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:50.453815 17365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:50.453846 17365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:50.455317 17365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:37827,127.3.246.125:35619,127.3.246.124:37623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:50.455652 17365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:50.455888 17365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:50.460595 17370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:50.460595 17373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:50.460623 17371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:50.460773 17365 server_base.cc:1034] running on GCE node
I20260924 07:20:50.461104 17365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:50.461393 17365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:50.462576 17365 hybrid_clock.cc:648] HybridClock initialized: now 1790234450462541 us; error 49 us; skew 500 ppm
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:50.465075 17365 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:20:50.465514 17365 webserver.cc:460] Webserver started at http://127.3.246.126:45597/ using document root <none> and password file <none>
I20260924 07:20:50.465813 17365 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:50.465885 17365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:50.466042 17365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:50.467216 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "aced3ea7dd594613a00ebcac460d8414"
format_stamp: "Formatted at 2026-09-24 07:20:50 on dist-test-slave-rc1w"
server_key: "4c1d0280ed7b258fbcec1f3c8db9b66c"
server_key_iv: "0d4d55948bd34c6a8919d1b1de5676f4"
server_key_version: "encryptionkey@0"
I20260924 07:20:50.467581 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "aced3ea7dd594613a00ebcac460d8414"
format_stamp: "Formatted at 2026-09-24 07:20:50 on dist-test-slave-rc1w"
server_key: "4c1d0280ed7b258fbcec1f3c8db9b66c"
server_key_iv: "0d4d55948bd34c6a8919d1b1de5676f4"
server_key_version: "encryptionkey@0"
I20260924 07:20:50.468926 17365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:20:50.469610 17380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:50.469812 17365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:20:50.469918 17365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "aced3ea7dd594613a00ebcac460d8414"
format_stamp: "Formatted at 2026-09-24 07:20:50 on dist-test-slave-rc1w"
server_key: "4c1d0280ed7b258fbcec1f3c8db9b66c"
server_key_iv: "0d4d55948bd34c6a8919d1b1de5676f4"
server_key_version: "encryptionkey@0"
I20260924 07:20:50.470002 17365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:50.480196 17365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:50.482121 17365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:50.482288 17365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:50.486173 17365 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:37827
I20260924 07:20:50.486207 17432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:37827 every 8 connection(s)
I20260924 07:20:50.486563 17365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:50.486987 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 17365
I20260924 07:20:50.487118 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:20:50.487308 4057 external_mini_cluster.cc:1428] Setting key 663728aac7510fa596c63516a7939c46
I20260924 07:20:50.487552 17433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:50.489498 17433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.246.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.125@KRBTEST.COM" created.
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:50.503341 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.490350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37827 (local address 127.0.0.1:33282)
0924 07:20:50.490555 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.490594 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:50.491135 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:50.491433 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:50.491436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:50.491719 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:50.491873 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.491877 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.492867 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.492869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.493532 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.493534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.494098 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.494099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.494165 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.494741 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:50.494750 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:50.498530 (+ 3780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:50.501712 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.501718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.501720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.501918 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.502175 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.502176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.502177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.502290 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.502738 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:50.502743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:50.502824 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 07:20:50.502917 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 07:20:50.503081 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
I20260924 07:20:50.503660 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.490553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.3.246.126:37827)
0924 07:20:50.490694 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.490699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.490732 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:20:50.491302 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.491304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.491328 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.491428 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.491986 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.492766 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.493634 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.494000 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.494126 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.498664 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.498673 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.498674 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.498682 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.501610 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.502017 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.502020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.502021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.502108 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.502362 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.502363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.502364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.502643 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.502752 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.503492 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.503571 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
I20260924 07:20:50.505530 17433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:50.505908 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.505781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35619 (local address 127.0.0.1:41318)
0924 07:20:50.505838 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.505888 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:35619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:20:50.506027 17381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.246.125:35619: connect: Connection refused (error 111)
I20260924 07:20:50.506422 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.506320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35619 (local address 127.0.0.1:41326)
0924 07:20:50.506377 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.506401 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:35619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 07:20:50.506525 17433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.125:35619: Network error: Client connection negotiation failed: client connection to 127.3.246.125:35619: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.125.keytab.
Entry for principal kudu/127.3.246.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.125.keytab.
I20260924 07:20:50.516273 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.125:35619
--webserver_interface=127.3.246.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.125.keytab
--principal=kudu/127.3.246.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:37827,127.3.246.125:35619,127.3.246.124:37623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:20:50.555764 17433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } attempt: 1
I20260924 07:20:50.556252 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.556112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35619 (local address 127.0.0.1:41332)
0924 07:20:50.556178 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.556217 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:35619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:20:50.556598 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.556500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35619 (local address 127.0.0.1:41346)
0924 07:20:50.556547 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.556572 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:35619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 07:20:50.556679 17433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.125:35619: Network error: Client connection negotiation failed: client connection to 127.3.246.125:35619: connect: Connection refused (error 111)
W20260924 07:20:50.595504 17440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:50.595727 17440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:50.595762 17440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:50.597347 17440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:50.597419 17440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:50.597445 17440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:50.597476 17440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:50.597500 17440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:50.598982 17440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:37827,127.3.246.125:35619,127.3.246.124:37623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.125.keytab
--principal=kudu/127.3.246.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.125:35619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.246.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:50.599246 17440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:50.599475 17440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:50.604131 17448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:50.604168 17445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:50.604250 17440 server_base.cc:1034] running on GCE node
W20260924 07:20:50.604143 17446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:50.604579 17440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:50.604854 17440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:50.605997 17440 hybrid_clock.cc:648] HybridClock initialized: now 1790234450605984 us; error 31 us; skew 500 ppm
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:50.608415 17440 init.cc:377] Logged in from keytab as kudu/127.3.246.125@KRBTEST.COM (short username kudu)
I20260924 07:20:50.608861 17440 webserver.cc:460] Webserver started at http://127.3.246.125:36067/ using document root <none> and password file <none>
I20260924 07:20:50.609098 17440 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:50.609164 17440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:50.609314 17440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:50.610390 17440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/data/instance:
uuid: "44ddb1b121cb4dc087bca47cdea0473e"
format_stamp: "Formatted at 2026-09-24 07:20:50 on dist-test-slave-rc1w"
server_key: "3139dfef94514d9387e8abb040e2834e"
server_key_iv: "45b3ce12a335ddb807cbc4994ba68c9f"
server_key_version: "encryptionkey@0"
I20260924 07:20:50.610759 17440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/wal/instance:
uuid: "44ddb1b121cb4dc087bca47cdea0473e"
format_stamp: "Formatted at 2026-09-24 07:20:50 on dist-test-slave-rc1w"
server_key: "3139dfef94514d9387e8abb040e2834e"
server_key_iv: "45b3ce12a335ddb807cbc4994ba68c9f"
server_key_version: "encryptionkey@0"
I20260924 07:20:50.612062 17440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:20:50.612802 17455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:50.612977 17440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:20:50.613066 17440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/wal
uuid: "44ddb1b121cb4dc087bca47cdea0473e"
format_stamp: "Formatted at 2026-09-24 07:20:50 on dist-test-slave-rc1w"
server_key: "3139dfef94514d9387e8abb040e2834e"
server_key_iv: "45b3ce12a335ddb807cbc4994ba68c9f"
server_key_version: "encryptionkey@0"
I20260924 07:20:50.613145 17440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:50.623742 17440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:50.624821 17433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } attempt: 2
I20260924 07:20:50.625303 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.625151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35619 (local address 127.0.0.1:41362)
0924 07:20:50.625218 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.625259 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:35619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:20:50.625694 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.625543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35619 (local address 127.0.0.1:41378)
0924 07:20:50.625576 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.625595 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:35619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 07:20:50.625797 17433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.125:35619: Network error: Client connection negotiation failed: client connection to 127.3.246.125:35619: connect: Connection refused (error 111)
I20260924 07:20:50.626123 17440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:50.626226 17440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:50.629964 17440 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.125:35619
I20260924 07:20:50.630013 17507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.125:35619 every 8 connection(s)
I20260924 07:20:50.630337 17440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/data/info.pb
I20260924 07:20:50.630949 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 17440
I20260924 07:20:50.631091 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-1/wal/instance
I20260924 07:20:50.631322 4057 external_mini_cluster.cc:1428] Setting key 1b13f5c5be7b67b9adc2819a6ac8a964
I20260924 07:20:50.631453 17508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:50.633365 17508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.246.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.124@KRBTEST.COM" created.
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.125@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:50.644860 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.634209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.3.246.126:37827)
0924 07:20:50.634277 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.634279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.634505 (+ 226us) server_negotiation.cc:407] Connection header received
0924 07:20:50.634987 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.634988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.635003 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.635052 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.635659 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.636147 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.637101 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.637317 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.637395 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.641959 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.641960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.641961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.641969 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.643397 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.643827 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.643829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.643829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.643894 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.644132 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.644133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.644133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.644289 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.644362 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.644613 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.644710 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:20:50.644887 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.634187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37827 (local address 127.0.0.1:33284)
0924 07:20:50.634368 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.634413 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:50.634878 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:50.635118 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:50.635121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:50.635359 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:50.635563 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.635567 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.636240 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.636241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.637009 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.637010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.637408 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.637409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.637529 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.638113 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:50.638122 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:50.641853 (+ 3731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:50.643480 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.643486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.643488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.643742 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.643972 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.643972 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.643973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.644065 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.644355 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:50.644359 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:50.644477 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 07:20:50.644558 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:20:50.644723 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":114,"threads_started":1}
I20260924 07:20:50.645566 17508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.125@KRBTEST.COM for kudu/127.3.246.125@KRBTEST.COM
I20260924 07:20:50.653579 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.645828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35619 (local address 127.0.0.1:41392)
0924 07:20:50.645876 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.645906 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:50.645947 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:50.646213 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:50.646214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:50.646351 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:50.646394 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.646396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.647246 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.647247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.647738 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.647739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.648189 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.648190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.648259 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.648663 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:50.648668 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:50.651035 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:50.652534 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.652534 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.652535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.652688 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.652880 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.652881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.652881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.652926 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.653250 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:50.653251 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:50.653291 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 07:20:50.653341 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 07:20:50.653439 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 07:20:50.653674 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.645917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.3.246.125:35619)
0924 07:20:50.646079 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.646082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.646097 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:20:50.646113 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.646115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.646124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.646198 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.646468 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.647147 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.647824 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.648107 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.648187 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.651120 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.651127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.651127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.651133 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.652455 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.652758 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.652760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.652761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.652813 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.653003 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.653004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.653004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.653187 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.653252 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.653371 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.653528 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260924 07:20:50.654277 17508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:50.654654 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.654505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:37623 (local address 127.0.0.1:54980)
0924 07:20:50.654576 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.654632 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 07:20:50.654781 17456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
I20260924 07:20:50.655169 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.655050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:37623 (local address 127.0.0.1:54994)
0924 07:20:50.655120 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.655145 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:20:50.655252 17508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:37623: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.124.keytab.
Entry for principal kudu/127.3.246.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.124.keytab.
I20260924 07:20:50.656755 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.124:37623
--webserver_interface=127.3.246.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.124.keytab
--principal=kudu/127.3.246.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:37827,127.3.246.125:35619,127.3.246.124:37623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:20:50.704463 17508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } attempt: 1
I20260924 07:20:50.704967 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.704823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:37623 (local address 127.0.0.1:55006)
0924 07:20:50.704892 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.704932 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 07:20:50.705302 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.705209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:37623 (local address 127.0.0.1:55014)
0924 07:20:50.705254 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.705278 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 07:20:50.705403 17508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:37623: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
I20260924 07:20:50.716940 17433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } attempt: 3
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for kudu/127.3.246.125@KRBTEST.COM
I20260924 07:20:50.724156 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.717276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35619 (local address 127.0.0.1:41398)
0924 07:20:50.717341 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.717355 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:50.717430 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:50.717587 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:50.717589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:50.717829 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:50.717898 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.717900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.718590 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.718592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.719130 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.719131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.719533 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.719534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.719615 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.720076 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:50.720081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:50.721323 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:50.723187 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.723188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.723188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.723375 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.723590 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.723590 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.723591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.723645 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.723924 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:50.723925 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:50.723963 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 07:20:50.724016 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:20:50.724090 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 07:20:50.724288 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.717361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.3.246.125:35619)
0924 07:20:50.717422 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.717425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.717436 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:20:50.717499 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.717500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.717512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.717583 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.717989 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.718495 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.719216 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.719443 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.719524 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.721416 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.721418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.721418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.721428 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.723086 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.723453 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.723454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.723455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.723520 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.723710 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.723711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.723711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.723855 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.723934 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.724061 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.724151 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:20:50.724643 17433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:50.725008 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.724874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:37623 (local address 127.0.0.1:55022)
0924 07:20:50.724947 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.724984 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 07:20:50.725442 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.725334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:37623 (local address 127.0.0.1:55030)
0924 07:20:50.725384 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.725421 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:20:50.725571 17433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:37623: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
W20260924 07:20:50.739077 17515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:50.739272 17515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:50.739295 17515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:50.740345 17515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:50.740379 17515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:50.740394 17515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:50.740406 17515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:50.740418 17515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:50.741950 17515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:37827,127.3.246.125:35619,127.3.246.124:37623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.124.keytab
--principal=kudu/127.3.246.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.124:37623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.246.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:50.742149 17515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:50.742311 17515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:50.747051 17520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:50.747046 17521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:50.747046 17523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:50.747128 17515 server_base.cc:1034] running on GCE node
I20260924 07:20:50.747485 17515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:50.747748 17515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:50.748909 17515 hybrid_clock.cc:648] HybridClock initialized: now 1790234450748878 us; error 48 us; skew 500 ppm
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:50.751394 17515 init.cc:377] Logged in from keytab as kudu/127.3.246.124@KRBTEST.COM (short username kudu)
I20260924 07:20:50.751828 17515 webserver.cc:460] Webserver started at http://127.3.246.124:40389/ using document root <none> and password file <none>
I20260924 07:20:50.752066 17515 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:50.752136 17515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:50.752288 17515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:50.753346 17515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/data/instance:
uuid: "50db6946d1954395b517dfa793c632ae"
format_stamp: "Formatted at 2026-09-24 07:20:50 on dist-test-slave-rc1w"
server_key: "65efe911766bbf0c68221dccfb9e763c"
server_key_iv: "86ec3e62b1ca54579e5e315a0e2a5e70"
server_key_version: "encryptionkey@0"
I20260924 07:20:50.753779 17515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/wal/instance:
uuid: "50db6946d1954395b517dfa793c632ae"
format_stamp: "Formatted at 2026-09-24 07:20:50 on dist-test-slave-rc1w"
server_key: "65efe911766bbf0c68221dccfb9e763c"
server_key_iv: "86ec3e62b1ca54579e5e315a0e2a5e70"
server_key_version: "encryptionkey@0"
I20260924 07:20:50.755069 17515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:20:50.755784 17530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:50.755945 17515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:20:50.756043 17515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/wal
uuid: "50db6946d1954395b517dfa793c632ae"
format_stamp: "Formatted at 2026-09-24 07:20:50 on dist-test-slave-rc1w"
server_key: "65efe911766bbf0c68221dccfb9e763c"
server_key_iv: "86ec3e62b1ca54579e5e315a0e2a5e70"
server_key_version: "encryptionkey@0"
I20260924 07:20:50.756126 17515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:50.756726 17433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } attempt: 1
I20260924 07:20:50.757108 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.756986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:37623 (local address 127.0.0.1:55046)
0924 07:20:50.757043 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.757080 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 07:20:50.757433 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.757339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:37623 (local address 127.0.0.1:55062)
0924 07:20:50.757385 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.757408 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 07:20:50.757530 17433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:37623: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
I20260924 07:20:50.768262 17515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:50.770139 17515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:50.770295 17515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:50.773542 17508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } attempt: 2
I20260924 07:20:50.774009 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.773874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:37623 (local address 127.0.0.1:55064)
0924 07:20:50.773929 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.773979 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 07:20:50.774422 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.774290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:37623 (local address 127.0.0.1:55072)
0924 07:20:50.774348 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.774384 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 07:20:50.774505 17515 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.124:37623
W20260924 07:20:50.774528 17508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:37623: Network error: Client connection negotiation failed: client connection to 127.3.246.124:37623: connect: Connection refused (error 111)
I20260924 07:20:50.774550 17582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.124:37623 every 8 connection(s)
I20260924 07:20:50.774917 17515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/data/info.pb
I20260924 07:20:50.775799 17583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:50.777319 17583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:50.781673 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 17515
I20260924 07:20:50.781817 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234301618289-4057-0/minicluster-data/master-2/wal/instance
I20260924 07:20:50.782043 4057 external_mini_cluster.cc:1428] Setting key 4fc5c33b5c419526420837e6d1b45c16
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.124@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:50.787043 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.778149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.3.246.126:37827)
0924 07:20:50.778208 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.778209 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.778436 (+ 227us) server_negotiation.cc:407] Connection header received
0924 07:20:50.778921 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.778922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.778934 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.778982 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.779594 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.780084 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.781053 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.781242 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.781335 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.783948 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.783949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.783950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.783957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.785459 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.785938 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.785939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.785940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.786006 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.786327 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.786328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.786329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.786457 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.786512 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.786904 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.786984 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:20:50.787272 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.778129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37827 (local address 127.0.0.1:33294)
0924 07:20:50.778310 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.778345 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:50.778813 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:50.779053 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:50.779057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:50.779308 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:50.779500 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.779504 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.780186 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.780187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.780959 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.780961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.781344 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.781345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.781478 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.782113 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:50.782123 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:50.783865 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:50.785539 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.785545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.785547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.785839 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.786082 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.786083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.786083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.786229 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.786529 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:50.786536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:50.786700 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 07:20:50.786878 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 07:20:50.787048 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260924 07:20:50.787832 17583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.124@KRBTEST.COM
I20260924 07:20:50.791422 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.782459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.3.246.124:37623)
0924 07:20:50.782611 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.782615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.782671 (+ 56us) server_negotiation.cc:407] Connection header received
0924 07:20:50.782734 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.782737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.782761 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.782839 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.783583 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.784508 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.785370 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.785822 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.785881 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.788402 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.788410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.788410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.788416 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.790153 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.790548 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.790551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.790552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.790610 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.790804 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.790805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.790806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.790988 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.791108 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.791257 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.791328 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.124@KRBTEST.COM for kudu/127.3.246.125@KRBTEST.COM
I20260924 07:20:50.795243 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.788117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.3.246.125:35619)
0924 07:20:50.788169 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.788172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.788180 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:20:50.788205 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.788206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.788215 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.788263 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.788531 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.789164 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.790155 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.790466 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.790530 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.792525 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.792526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.792526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.792532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.794123 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.794490 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.794491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.794492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.794551 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.794749 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.794750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.794750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.794868 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.794942 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.795090 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.795157 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:20:50.795241 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.788056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:35619 (local address 127.0.0.1:41404)
0924 07:20:50.788101 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.788111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:50.788148 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:50.788263 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:50.788264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:50.788405 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:50.788463 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.788464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.789247 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.789247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.790059 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.790061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.790538 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.790539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.790611 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.791002 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:50.791006 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:50.792453 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:50.794228 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.794229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.794230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.794402 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.794619 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.794620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.794621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.794676 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.794935 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:50.794936 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:50.794977 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 07:20:50.795065 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:20:50.795152 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 07:20:50.795708 17583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:50.799369 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.792225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.3.246.126:37827)
0924 07:20:50.792265 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.792267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.792321 (+ 54us) server_negotiation.cc:407] Connection header received
0924 07:20:50.792391 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.792392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.792399 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.792440 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.792912 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.793427 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.794160 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.794358 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.794447 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.796713 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.796714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.796714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.796721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.798166 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.798595 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.798596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.798596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.798671 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.798874 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.798875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.798875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.798998 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.799194 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.799235 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.799303 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.124@KRBTEST.COM for kudu/127.3.246.124@KRBTEST.COM
I20260924 07:20:50.802739 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.796427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:37623 (local address 127.0.0.1:55094)
0924 07:20:50.796476 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.796487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:50.796538 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:50.796654 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:50.796655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:50.796778 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:50.796800 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.796801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.797417 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.797419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.797945 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.797947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.798322 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.798323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.798370 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.798717 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:50.798722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:50.800136 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:50.801746 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.801747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.801747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.801920 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.802111 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.802111 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.802111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.802158 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.802417 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:50.802418 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:50.802458 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 07:20:50.802534 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:20:50.802632 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 07:20:50.802937 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.796464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55094 (local address 127.3.246.124:37623)
0924 07:20:50.796494 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.796496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.796523 (+ 27us) server_negotiation.cc:407] Connection header received
0924 07:20:50.796591 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.796592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.796601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.796643 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.796881 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.797340 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.798032 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.798234 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.798318 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.800213 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.800214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.800214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.800221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.801638 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.801989 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.801990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.801990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.802047 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.802220 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.802221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.802222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.802346 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.802409 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.802550 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.802693 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":16}
I20260924 07:20:50.804455 17583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae: Bootstrap starting.
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.125@KRBTEST.COM
I20260924 07:20:50.805322 17583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:50.805615 17583 log.cc:824] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:50.806447 17583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae: No bootstrap required, opened a new log
I20260924 07:20:50.807673 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.799966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.3.246.125:35619)
0924 07:20:50.800000 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.800004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.800055 (+ 51us) server_negotiation.cc:407] Connection header received
0924 07:20:50.800128 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.800129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.800135 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.800176 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.800771 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.801359 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.802289 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.802539 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.802600 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.805004 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.805006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.805006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.805013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.806571 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.806917 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.806918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.806918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.806971 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.807173 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.807174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.807174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.807283 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.807478 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.807517 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.807585 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:20:50.808012 17583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } }
I20260924 07:20:50.808184 17583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:50.808223 17583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50db6946d1954395b517dfa793c632ae, State: Initialized, Role: FOLLOWER
I20260924 07:20:50.808365 17583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } }
I20260924 07:20:50.808722 17589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } } }
I20260924 07:20:50.808784 17583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:50.808799 17589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:50.809038 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 07:20:50.810586 17603 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:50.810698 17603 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:50.810968 17515 master_runner.cc:186] Error getting master registration for 127.3.246.126:37827: OK, instance_id { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" instance_seqno: 1790234450485025 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:20:50.812805 17515 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:20:50.818578 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.810364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.3.246.126:37827)
0924 07:20:50.810471 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.810473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.810517 (+ 44us) server_negotiation.cc:407] Connection header received
0924 07:20:50.810627 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.810628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.810638 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.810669 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.811423 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.812107 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.813072 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.813332 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.813373 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.815674 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.815676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.815676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.815683 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.817323 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.817742 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.817743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.817744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.817799 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.818010 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.818011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.818011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.818137 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.818205 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.818404 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.818484 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 07:20:50.820320 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.810948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.3.246.124:37623)
0924 07:20:50.810988 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.810990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.813093 (+ 2103us) server_negotiation.cc:407] Connection header received
0924 07:20:50.813184 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.813185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.813194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.813223 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.813939 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.814606 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.815528 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.815809 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.815849 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.817383 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.817385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.817385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.817392 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.819047 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.819464 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.819465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.819466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.819523 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.819719 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.819720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.819720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.819889 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.820010 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.820149 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.820228 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:20:50.821839 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.810818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.3.246.125:35619)
0924 07:20:50.810856 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.810858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.812926 (+ 2068us) server_negotiation.cc:407] Connection header received
0924 07:20:50.812998 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.812999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.813009 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.813062 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.814090 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.814521 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.815528 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.815849 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.816433 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.818671 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.818672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.818673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.818679 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.820371 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.820714 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.820715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.820715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.820770 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.820987 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.820988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.820988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.821089 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.821634 (+ 545us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.821711 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.821779 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:20:50.832677 17433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } attempt: 2
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for kudu/127.3.246.124@KRBTEST.COM
I20260924 07:20:50.839798 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.833070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55112 (local address 127.3.246.124:37623)
0924 07:20:50.833139 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.833141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.833150 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:20:50.833168 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.833169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.833177 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.833232 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.833532 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.834058 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.835038 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.835209 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.835263 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.837330 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.837332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.837332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.837338 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.838714 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.839092 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.839093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.839094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.839148 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.839383 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.839383 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.839383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.839475 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.839526 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.839689 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.839741 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:20:50.839843 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.832992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:37623 (local address 127.0.0.1:55112)
0924 07:20:50.833052 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.833064 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:50.833112 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:50.833250 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:50.833251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:50.833415 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:50.833462 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.833463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.834130 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.834131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.834962 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.834964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.835281 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.835283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.835338 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.835974 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:50.835978 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:50.837224 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:50.838812 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.838843 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.838843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.839006 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.839244 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.839245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.839245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.839315 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.839539 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:50.839540 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:50.839584 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 07:20:50.839661 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:20:50.839746 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 07:20:50.841295 17433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414: Bootstrap starting.
I20260924 07:20:50.841992 17433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:50.842288 17433 log.cc:824] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:50.843014 17433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414: No bootstrap required, opened a new log
I20260924 07:20:50.844398 17433 raft_consensus.cc:348] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } }
I20260924 07:20:50.844527 17433 raft_consensus.cc:374] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:50.844573 17433 raft_consensus.cc:729] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aced3ea7dd594613a00ebcac460d8414, State: Initialized, Role: FOLLOWER
I20260924 07:20:50.844718 17433 consensus_queue.cc:260] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } }
I20260924 07:20:50.845001 17610 sys_catalog.cc:455] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } } }
I20260924 07:20:50.845065 17433 sys_catalog.cc:564] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:50.845077 17610 sys_catalog.cc:458] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:20:50.846781 17621 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:50.846833 17621 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:50.847466 17365 master_runner.cc:186] Error getting master registration for 127.3.246.125:35619: OK, instance_id { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" instance_seqno: 1790234450628890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:20:50.850335 17365 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:20:50.865690 17508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } attempt: 3
Sep 24 07:20:50 dist-test-slave-rc1w krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.125@KRBTEST.COM for kudu/127.3.246.124@KRBTEST.COM
I20260924 07:20:50.874137 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.866004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:37623 (local address 127.0.0.1:55126)
0924 07:20:50.866059 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:50.866070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:50.866108 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:50.866263 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:50.866264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:50.866475 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:50.866529 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.866531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.867464 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.867466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.868308 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:50.868311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.868974 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.868975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:50.869038 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.869552 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:50.869557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:50.871068 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:50.872971 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.872973 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.872973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.873192 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.873426 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:50.873427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:50.873427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:50.873497 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:50.873797 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:50.873798 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:50.873847 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 07:20:50.873942 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 07:20:50.874037 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 07:20:50.874151 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:50.866070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.3.246.124:37623)
0924 07:20:50.866112 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:50.866114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:50.866152 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:20:50.866178 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:50.866178 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:50.866189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:50.866243 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:50.866630 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.867375 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.868415 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:50.868718 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:50.868777 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:50.871156 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:50.871157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:50.871157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:50.871165 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:50.872890 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.873258 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.873260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.873260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.873342 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:50.873567 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:50.873568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:50.873569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:50.873723 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:50.873795 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:50.873984 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 07:20:50.874059 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:20:50.875991 17508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e: Bootstrap starting.
I20260924 07:20:50.876736 17508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:50.877028 17508 log.cc:824] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:50.877986 17508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e: No bootstrap required, opened a new log
I20260924 07:20:50.880115 17508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } }
I20260924 07:20:50.880250 17508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:50.880285 17508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ddb1b121cb4dc087bca47cdea0473e, State: Initialized, Role: FOLLOWER
I20260924 07:20:50.880409 17508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } }
I20260924 07:20:50.880712 17508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [sys.catalog]: configured and running, proceeding with master startup.
W20260924 07:20:50.882462 17635 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:50.882561 17635 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:50.884305 17440 master_runner.cc:263] No leader master found from master 44ddb1b121cb4dc087bca47cdea0473e
I20260924 07:20:50.884367 17440 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:20:50.880671 17624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } } }
I20260924 07:20:50.884414 17624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:20:50.983058 17386 tablet.cc:2369] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:20:51.020947 17536 tablet.cc:2369] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:20:51.083859 17610 raft_consensus.cc:482] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:51.083986 17610 raft_consensus.cc:504] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } }
I20260924 07:20:51.084281 17610 leader_election.cc:288] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44ddb1b121cb4dc087bca47cdea0473e (127.3.246.125:35619), 50db6946d1954395b517dfa793c632ae (127.3.246.124:37623)
I20260924 07:20:51.084601 17557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aced3ea7dd594613a00ebcac460d8414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50db6946d1954395b517dfa793c632ae" is_pre_election: true
I20260924 07:20:51.084619 17482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aced3ea7dd594613a00ebcac460d8414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ddb1b121cb4dc087bca47cdea0473e" is_pre_election: true
I20260924 07:20:51.084767 17482 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aced3ea7dd594613a00ebcac460d8414 in term 0.
I20260924 07:20:51.084766 17557 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aced3ea7dd594613a00ebcac460d8414 in term 0.
I20260924 07:20:51.084959 17381 leader_election.cc:302] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50db6946d1954395b517dfa793c632ae, aced3ea7dd594613a00ebcac460d8414; no voters:
I20260924 07:20:51.085139 17610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:20:51.085206 17610 raft_consensus.cc:482] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:20:51.085223 17610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:51.086051 17610 raft_consensus.cc:504] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } }
I20260924 07:20:51.086180 17610 leader_election.cc:288] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [CANDIDATE]: Term 1 election: Requested vote from peers 44ddb1b121cb4dc087bca47cdea0473e (127.3.246.125:35619), 50db6946d1954395b517dfa793c632ae (127.3.246.124:37623)
I20260924 07:20:51.086375 17482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aced3ea7dd594613a00ebcac460d8414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ddb1b121cb4dc087bca47cdea0473e"
I20260924 07:20:51.086436 17557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aced3ea7dd594613a00ebcac460d8414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50db6946d1954395b517dfa793c632ae"
I20260924 07:20:51.086467 17482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:51.086539 17557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:51.087182 17482 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aced3ea7dd594613a00ebcac460d8414 in term 1.
I20260924 07:20:51.087334 17557 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aced3ea7dd594613a00ebcac460d8414 in term 1.
I20260924 07:20:51.087365 17381 leader_election.cc:302] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ddb1b121cb4dc087bca47cdea0473e, aced3ea7dd594613a00ebcac460d8414; no voters:
I20260924 07:20:51.087443 17610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:51.087548 17610 raft_consensus.cc:686] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [term 1 LEADER]: Becoming Leader. State: Replica: aced3ea7dd594613a00ebcac460d8414, State: Running, Role: LEADER
I20260924 07:20:51.087633 17610 consensus_queue.cc:237] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } }
I20260924 07:20:51.088022 17637 sys_catalog.cc:455] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aced3ea7dd594613a00ebcac460d8414. Latest consensus state: current_term: 1 leader_uuid: "aced3ea7dd594613a00ebcac460d8414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } } }
I20260924 07:20:51.088116 17637 sys_catalog.cc:458] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:51.088338 17639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:51.088536 17639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:51.089910 17482 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [term 1 FOLLOWER]: Refusing update from remote peer aced3ea7dd594613a00ebcac460d8414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:51.089984 17557 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [term 1 FOLLOWER]: Refusing update from remote peer aced3ea7dd594613a00ebcac460d8414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:51.090206 17637 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:51.090328 17610 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:51.091477 17589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader aced3ea7dd594613a00ebcac460d8414. Latest consensus state: current_term: 1 leader_uuid: "aced3ea7dd594613a00ebcac460d8414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } } }
I20260924 07:20:51.091472 17624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [sys.catalog]: SysCatalogTable state changed. Reason: New leader aced3ea7dd594613a00ebcac460d8414. Latest consensus state: current_term: 1 leader_uuid: "aced3ea7dd594613a00ebcac460d8414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } } }
I20260924 07:20:51.091557 17624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:51.091557 17589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:51.092338 17610 sys_catalog.cc:455] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aced3ea7dd594613a00ebcac460d8414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } } }
I20260924 07:20:51.092363 17637 sys_catalog.cc:455] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aced3ea7dd594613a00ebcac460d8414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } } }
I20260924 07:20:51.092427 17610 sys_catalog.cc:458] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:51.092435 17637 sys_catalog.cc:458] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:51.092581 17641 mvcc.cc:205] Tried to move back new op lower bound from 7332800311661912064 to 7332800311655596032. Current Snapshot: MvccSnapshot[applied={T|T < 7332800311661912064}]
I20260924 07:20:51.092618 17624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aced3ea7dd594613a00ebcac460d8414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } } }
I20260924 07:20:51.092667 17589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aced3ea7dd594613a00ebcac460d8414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aced3ea7dd594613a00ebcac460d8414" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37827 } } peers { permanent_uuid: "44ddb1b121cb4dc087bca47cdea0473e" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 35619 } } peers { permanent_uuid: "50db6946d1954395b517dfa793c632ae" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 37623 } } }
I20260924 07:20:51.092703 17624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:51.092725 17589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:51.093098 17642 mvcc.cc:205] Tried to move back new op lower bound from 7332800311661912064 to 7332800311655596032. Current Snapshot: MvccSnapshot[applied={T|T < 7332800311661912064 or (T in {7332800311661912064})}]
I20260924 07:20:51.093200 17639 catalog_manager.cc:1347] Generated new cluster ID: d12caef83d1844e0ae56e0409bb97c76
I20260924 07:20:51.093245 17639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:51.102881 17639 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:51.103368 17639 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:51.107344 17639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aced3ea7dd594613a00ebcac460d8414: Generated new TSK 0
I20260924 07:20:51.107512 17639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:51.811425 17603 catalog_manager.cc:1259] Loaded cluster ID: d12caef83d1844e0ae56e0409bb97c76
I20260924 07:20:51.811499 17603 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae: loading cluster ID for follower catalog manager: success
I20260924 07:20:51.812291 17603 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae: acquiring CA information for follower catalog manager: success
I20260924 07:20:51.812546 17603 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50db6946d1954395b517dfa793c632ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:20:51.883023 17635 catalog_manager.cc:1259] Loaded cluster ID: d12caef83d1844e0ae56e0409bb97c76
I20260924 07:20:51.883101 17635 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e: loading cluster ID for follower catalog manager: success
I20260924 07:20:51.883850 17635 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e: acquiring CA information for follower catalog manager: success
I20260924 07:20:51.884116 17635 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44ddb1b121cb4dc087bca47cdea0473e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:20:52.164844 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 17365
I20260924 07:20:52.170192 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 17440
I20260924 07:20:52.175184 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 17515
2026-09-24T07:20:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2953 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:52 dist-test-slave-rc1w krb5kdc[17655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:52 dist-test-slave-rc1w krb5kdc[17655](info): set up 2 sockets
Sep 24 07:20:52 dist-test-slave-rc1w krb5kdc[17655](info): commencing operation
krb5kdc: starting...
W20260924 07:20:53.225792 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:20:53 dist-test-slave-rc1w krb5kdc[17655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:20:53.344657 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:34871
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:34871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:53.423530 17668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:53.423729 17668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:53.423751 17668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:53.424816 17668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:53.424850 17668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:53.424865 17668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:53.424877 17668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:53.424889 17668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:53.426348 17668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:34871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:34871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:53.426537 17668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:53.426692 17668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:53.431569 17676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:53.431649 17668 server_base.cc:1034] running on GCE node
W20260924 07:20:53.431589 17673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:53.431586 17674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:53.432070 17668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:53.432382 17668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:53.433548 17668 hybrid_clock.cc:648] HybridClock initialized: now 1790234453433531 us; error 39 us; skew 500 ppm
Sep 24 07:20:53 dist-test-slave-rc1w krb5kdc[17655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:53.435958 17668 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:20:53.436333 17668 webserver.cc:460] Webserver started at http://127.3.246.126:43445/ using document root <none> and password file <none>
I20260924 07:20:53.436518 17668 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:53.436558 17668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:53.436762 17668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:53.437896 17668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "cc2fbe28a07e47028ca48f522da9c248"
format_stamp: "Formatted at 2026-09-24 07:20:53 on dist-test-slave-rc1w"
server_key: "5fb86efedb0d2c97b5ee44824381bae0"
server_key_iv: "50f0ba5f8bd281867dc76267dc8e9992"
server_key_version: "encryptionkey@0"
I20260924 07:20:53.438266 17668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "cc2fbe28a07e47028ca48f522da9c248"
format_stamp: "Formatted at 2026-09-24 07:20:53 on dist-test-slave-rc1w"
server_key: "5fb86efedb0d2c97b5ee44824381bae0"
server_key_iv: "50f0ba5f8bd281867dc76267dc8e9992"
server_key_version: "encryptionkey@0"
I20260924 07:20:53.439627 17668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:20:53.440445 17683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:53.440659 17668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:53.440761 17668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "cc2fbe28a07e47028ca48f522da9c248"
format_stamp: "Formatted at 2026-09-24 07:20:53 on dist-test-slave-rc1w"
server_key: "5fb86efedb0d2c97b5ee44824381bae0"
server_key_iv: "50f0ba5f8bd281867dc76267dc8e9992"
server_key_version: "encryptionkey@0"
I20260924 07:20:53.440850 17668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:53.451936 17668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:53.453889 17668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:53.454044 17668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:53.457742 17668 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:34871
I20260924 07:20:53.457794 17735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:34871 every 8 connection(s)
I20260924 07:20:53.458141 17668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:53.459074 17736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:53.459075 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 17668
I20260924 07:20:53.459168 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:20:53.459374 4057 external_mini_cluster.cc:1428] Setting key 759244d4f12706bd9fc46ea869ab90ca
I20260924 07:20:53.462370 17736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248: Bootstrap starting.
I20260924 07:20:53.463089 17736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:53.463382 17736 log.cc:824] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:53.464087 17736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248: No bootstrap required, opened a new log
Sep 24 07:20:53 dist-test-slave-rc1w krb5kdc[17655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:53.466109 17736 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2fbe28a07e47028ca48f522da9c248" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34871 } }
I20260924 07:20:53.466270 17736 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:53.466325 17736 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2fbe28a07e47028ca48f522da9c248, State: Initialized, Role: FOLLOWER
I20260924 07:20:53.466470 17736 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2fbe28a07e47028ca48f522da9c248" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34871 } }
I20260924 07:20:53.466549 17736 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:53.466571 17736 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:53.466637 17736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:53.467326 17736 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2fbe28a07e47028ca48f522da9c248" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34871 } }
I20260924 07:20:53.467447 17736 leader_election.cc:302] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc2fbe28a07e47028ca48f522da9c248; no voters:
I20260924 07:20:53.467625 17736 leader_election.cc:288] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:53.467648 17741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:53.467840 17741 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [term 1 LEADER]: Becoming Leader. State: Replica: cc2fbe28a07e47028ca48f522da9c248, State: Running, Role: LEADER
I20260924 07:20:53.467898 17736 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:53.467986 17741 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2fbe28a07e47028ca48f522da9c248" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34871 } }
I20260924 07:20:53.468372 17741 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc2fbe28a07e47028ca48f522da9c248. Latest consensus state: current_term: 1 leader_uuid: "cc2fbe28a07e47028ca48f522da9c248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2fbe28a07e47028ca48f522da9c248" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34871 } } }
I20260924 07:20:53.468396 17743 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc2fbe28a07e47028ca48f522da9c248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2fbe28a07e47028ca48f522da9c248" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34871 } } }
I20260924 07:20:53.468508 17741 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:53.468516 17743 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:53.468786 17750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:53.469345 17750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:53.470613 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:53.459893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.3.246.126:34871)
0924 07:20:53.460054 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:53.460061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:53.460085 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:20:53.460532 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:53.460539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:53.460583 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:53.460693 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:53.461519 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:53.462539 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:53.463619 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:53.464147 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:53.464262 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:53.466729 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:53.466738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:53.466738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:53.466747 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:53.468747 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:53.469133 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:53.469136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:53.469137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:53.469246 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:53.469485 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:53.469486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:53.469486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:53.469790 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:53.469928 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:53.470276 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 07:20:53.470415 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260924 07:20:53.471076 17750 catalog_manager.cc:1347] Generated new cluster ID: 35c0431cb79e4f1eac0f13ccb1d74431
I20260924 07:20:53.471220 17750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:53.471575 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:53.475850 17750 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:53.476539 17750 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:53.488598 17750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc2fbe28a07e47028ca48f522da9c248: Generated new TSK 0
I20260924 07:20:53.488806 17750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:53.525475 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 17668
2026-09-24T07:20:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1349 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:53 dist-test-slave-rc1w krb5kdc[17763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:53 dist-test-slave-rc1w krb5kdc[17763](info): set up 2 sockets
Sep 24 07:20:53 dist-test-slave-rc1w krb5kdc[17763](info): commencing operation
krb5kdc: starting...
W20260924 07:20:54.553841 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:20:54 dist-test-slave-rc1w krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:20:54.673823 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:43111
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:43111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:54.755103 17776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:54.755332 17776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:54.755380 17776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:54.756529 17776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:54.756572 17776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:54.756587 17776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:54.756598 17776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:54.756610 17776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:54.758451 17776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:43111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:43111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:54.758705 17776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:54.758939 17776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:54.763698 17782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:54.763726 17781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:54.763936 17776 server_base.cc:1034] running on GCE node
W20260924 07:20:54.763826 17784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:54.764256 17776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:54.764547 17776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:54.765763 17776 hybrid_clock.cc:648] HybridClock initialized: now 1790234454765758 us; error 41 us; skew 500 ppm
Sep 24 07:20:54 dist-test-slave-rc1w krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:54.768527 17776 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:20:54.768901 17776 webserver.cc:460] Webserver started at http://127.3.246.126:45081/ using document root <none> and password file <none>
I20260924 07:20:54.769090 17776 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:54.769129 17776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:54.769210 17776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:54.770244 17776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "1305b1d1f3a74376abb0278f7860a18d"
format_stamp: "Formatted at 2026-09-24 07:20:54 on dist-test-slave-rc1w"
server_key: "ad40f7e39b6abc2d78f6754a43584563"
server_key_iv: "c88529992ed1c161988c328f9d2dd9e0"
server_key_version: "encryptionkey@0"
I20260924 07:20:54.770533 17776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "1305b1d1f3a74376abb0278f7860a18d"
format_stamp: "Formatted at 2026-09-24 07:20:54 on dist-test-slave-rc1w"
server_key: "ad40f7e39b6abc2d78f6754a43584563"
server_key_iv: "c88529992ed1c161988c328f9d2dd9e0"
server_key_version: "encryptionkey@0"
I20260924 07:20:54.771695 17776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:20:54.772396 17791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:54.772575 17776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:54.772662 17776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "1305b1d1f3a74376abb0278f7860a18d"
format_stamp: "Formatted at 2026-09-24 07:20:54 on dist-test-slave-rc1w"
server_key: "ad40f7e39b6abc2d78f6754a43584563"
server_key_iv: "c88529992ed1c161988c328f9d2dd9e0"
server_key_version: "encryptionkey@0"
I20260924 07:20:54.772728 17776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:54.782059 17776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:54.783840 17776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:54.784017 17776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:54.787632 17776 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:43111
I20260924 07:20:54.787683 17843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:43111 every 8 connection(s)
I20260924 07:20:54.788002 17776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:54.788511 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 17776
I20260924 07:20:54.788653 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:20:54.788892 4057 external_mini_cluster.cc:1428] Setting key 876addc9b140960752dc5f6069726f49
I20260924 07:20:54.789178 17844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:54.791888 17844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Bootstrap starting.
I20260924 07:20:54.792454 17844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:54.792722 17844 log.cc:824] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:54.793368 17844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: No bootstrap required, opened a new log
Sep 24 07:20:54 dist-test-slave-rc1w krb5kdc[17763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:54.795136 17844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } }
I20260924 07:20:54.795274 17844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:54.795324 17844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1305b1d1f3a74376abb0278f7860a18d, State: Initialized, Role: FOLLOWER
I20260924 07:20:54.795459 17844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } }
I20260924 07:20:54.795540 17844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:54.795580 17844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:54.795629 17844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:54.796151 17844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } }
I20260924 07:20:54.796254 17844 leader_election.cc:302] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1305b1d1f3a74376abb0278f7860a18d; no voters:
I20260924 07:20:54.796446 17844 leader_election.cc:288] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:54.796473 17849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:54.796579 17849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 1 LEADER]: Becoming Leader. State: Replica: 1305b1d1f3a74376abb0278f7860a18d, State: Running, Role: LEADER
I20260924 07:20:54.796705 17849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } }
I20260924 07:20:54.796705 17844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:54.797096 17850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1305b1d1f3a74376abb0278f7860a18d. Latest consensus state: current_term: 1 leader_uuid: "1305b1d1f3a74376abb0278f7860a18d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } } }
I20260924 07:20:54.797240 17850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:54.797899 17851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1305b1d1f3a74376abb0278f7860a18d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } } }
I20260924 07:20:54.797986 17851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:54.798194 17855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:54.798748 17855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:54.799567 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:54.789342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.3.246.126:43111)
0924 07:20:54.789524 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:54.789534 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:54.789569 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:20:54.790280 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:54.790287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:54.790333 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:54.790476 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:54.791082 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:54.791922 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:54.792746 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:54.793306 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:54.793429 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:54.795787 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:54.795790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:54.795791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:54.795802 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:54.797833 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:54.798281 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:54.798284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:54.798284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:54.798387 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:54.798618 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:54.798619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:54.798620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:54.798927 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:54.799064 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:54.799234 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 07:20:54.799375 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260924 07:20:54.800917 17855 catalog_manager.cc:1347] Generated new cluster ID: 22e3036aea534bcda916c64eb1afc287
I20260924 07:20:54.800958 17855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:54.810606 17855 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:54.811101 17855 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:54.817168 17855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Generated new TSK 0
I20260924 07:20:54.817330 17855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:20:54.842661 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:43111
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:54.923637 17870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:54.923827 17870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:54.923851 17870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:54.924970 17870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:54.925012 17870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:54.925038 17870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:20:54.926564 17870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:54.926803 17870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:54.927030 17870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:54.931898 17878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:54.931923 17876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:54.931923 17875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:54.932180 17870 server_base.cc:1034] running on GCE node
I20260924 07:20:54.932364 17870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:54.932554 17870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:54.933715 17870 hybrid_clock.cc:648] HybridClock initialized: now 1790234454933689 us; error 40 us; skew 500 ppm
Sep 24 07:20:54 dist-test-slave-rc1w krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:54.936137 17870 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:20:54.936545 17870 webserver.cc:460] Webserver started at http://127.3.246.65:36929/ using document root <none> and password file <none>
I20260924 07:20:54.936771 17870 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:54.936815 17870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:54.936899 17870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:54.937927 17870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "651bd68fbf3d496c9dc50e8af8461e66"
format_stamp: "Formatted at 2026-09-24 07:20:54 on dist-test-slave-rc1w"
server_key: "bc3788d2c75c8a70e639680238efc587"
server_key_iv: "a553ea6bd34c46427b73b130c75a0b2f"
server_key_version: "encryptionkey@0"
I20260924 07:20:54.938218 17870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "651bd68fbf3d496c9dc50e8af8461e66"
format_stamp: "Formatted at 2026-09-24 07:20:54 on dist-test-slave-rc1w"
server_key: "bc3788d2c75c8a70e639680238efc587"
server_key_iv: "a553ea6bd34c46427b73b130c75a0b2f"
server_key_version: "encryptionkey@0"
I20260924 07:20:54.939391 17870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:54.940006 17885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:54.940160 17870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:20:54.940225 17870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "651bd68fbf3d496c9dc50e8af8461e66"
format_stamp: "Formatted at 2026-09-24 07:20:54 on dist-test-slave-rc1w"
server_key: "bc3788d2c75c8a70e639680238efc587"
server_key_iv: "a553ea6bd34c46427b73b130c75a0b2f"
server_key_version: "encryptionkey@0"
I20260924 07:20:54.940274 17870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:54.955788 17870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:54.958027 17870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:54.958133 17870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:54.958290 17870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:54.958523 17870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:54.958554 17870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:54.958575 17870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:54.958587 17870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:54.964165 17870 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:46021
I20260924 07:20:54.964213 17998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:46021 every 8 connection(s)
I20260924 07:20:54.964498 17870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:20:54.967099 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 17870
I20260924 07:20:54.967219 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:20:54.967412 4057 external_mini_cluster.cc:1428] Setting key 961da2f8ed76a05acc13422812c5efad
Sep 24 07:20:54 dist-test-slave-rc1w krb5kdc[17763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:54.974866 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:54.965187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:34129 (local address 127.3.246.126:43111)
0924 07:20:54.965230 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:54.965232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:54.965408 (+ 176us) server_negotiation.cc:407] Connection header received
0924 07:20:54.966008 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:54.966009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:54.966021 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:54.966071 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:54.967028 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:54.967675 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:54.969191 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:54.969389 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:54.969484 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:54.971617 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:54.971619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:54.971619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:54.971627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:54.973351 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:54.973835 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:54.973836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:54.973837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:54.973903 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:54.974175 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:54.974177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:54.974177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:54.974332 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:54.974403 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:54.974653 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 07:20:54.974751 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:20:54.975003 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:54.965146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:34129)
0924 07:20:54.965279 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:54.965328 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:54.965902 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:54.966145 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:54.966149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:54.966713 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:54.966897 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:54.966920 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:54.967763 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:54.967764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:54.969106 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:54.969109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:54.969492 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:54.969494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:54.969632 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:54.970167 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:54.970180 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:54.971523 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:54.973455 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:54.973461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:54.973463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:54.973738 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:54.973990 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:54.973991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:54.973992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:54.974095 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:54.974398 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:54.974403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:54.974513 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 07:20:54.974595 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 07:20:54.974710 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260924 07:20:54.975533 17999 heartbeater.cc:347] Connected to a master server at 127.3.246.126:43111
I20260924 07:20:54.975638 17999 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:54.975828 17999 heartbeater.cc:511] Master 127.3.246.126:43111 requested a full tablet report, sending...
I20260924 07:20:54.976428 17808 ts_manager.cc:194] Registered new tserver with Master: 651bd68fbf3d496c9dc50e8af8461e66 (127.3.246.65:46021)
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
I20260924 07:20:54.977241 17808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:34129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:20:54.992746 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:43111
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:55.072764 18004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:55.072968 18004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:55.072991 18004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:55.074086 18004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:55.074119 18004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:55.074144 18004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:20:55.075506 18004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:55.075789 18004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:55.076018 18004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:55.080901 18009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:55.081010 18010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:55.081019 18004 server_base.cc:1034] running on GCE node
W20260924 07:20:55.080912 18012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:55.081403 18004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:55.081602 18004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:55.082734 18004 hybrid_clock.cc:648] HybridClock initialized: now 1790234455082722 us; error 32 us; skew 500 ppm
Sep 24 07:20:55 dist-test-slave-rc1w krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:55.085227 18004 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:20:55.085733 18004 webserver.cc:460] Webserver started at http://127.3.246.66:35095/ using document root <none> and password file <none>
I20260924 07:20:55.085958 18004 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:55.086004 18004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:55.086138 18004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:55.087136 18004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "e24f48a3e33449ed9d349aa731765fc4"
format_stamp: "Formatted at 2026-09-24 07:20:55 on dist-test-slave-rc1w"
server_key: "03d2f076bee418ce3e2bc20f2abc0481"
server_key_iv: "54868eb8b1d9ae3fe489a876a8a9be81"
server_key_version: "encryptionkey@0"
I20260924 07:20:55.087478 18004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "e24f48a3e33449ed9d349aa731765fc4"
format_stamp: "Formatted at 2026-09-24 07:20:55 on dist-test-slave-rc1w"
server_key: "03d2f076bee418ce3e2bc20f2abc0481"
server_key_iv: "54868eb8b1d9ae3fe489a876a8a9be81"
server_key_version: "encryptionkey@0"
I20260924 07:20:55.088781 18004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:20:55.089535 18019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.089749 18004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:20:55.089843 18004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "e24f48a3e33449ed9d349aa731765fc4"
format_stamp: "Formatted at 2026-09-24 07:20:55 on dist-test-slave-rc1w"
server_key: "03d2f076bee418ce3e2bc20f2abc0481"
server_key_iv: "54868eb8b1d9ae3fe489a876a8a9be81"
server_key_version: "encryptionkey@0"
I20260924 07:20:55.089916 18004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:55.103255 18004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:55.105065 18004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:55.105211 18004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:55.105424 18004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:55.105784 18004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:55.105819 18004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.105841 18004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:55.105906 18004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.111537 18004 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:35815
I20260924 07:20:55.111578 18132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:35815 every 8 connection(s)
I20260924 07:20:55.111894 18004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:20:55.117175 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 18004
I20260924 07:20:55.117301 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:20:55.117513 4057 external_mini_cluster.cc:1428] Setting key 29f8da5c94ce32e41401e82500962eab
Sep 24 07:20:55 dist-test-slave-rc1w krb5kdc[17763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:55.121325 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.112577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:44363 (local address 127.3.246.126:43111)
0924 07:20:55.112636 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.112639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.112828 (+ 189us) server_negotiation.cc:407] Connection header received
0924 07:20:55.113323 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.113324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.113333 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.113372 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:55.114278 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.114782 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.115822 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.116024 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.116088 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.118527 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:55.118529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:55.118529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:55.118537 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:55.119974 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.120384 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.120385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.120386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.120445 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.120694 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.120695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.120695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.120808 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:55.120874 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.121143 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.121224 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:20:55.121516 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.112560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:44363)
0924 07:20:55.112705 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.112744 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:55.113227 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:55.113432 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:55.113435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:55.114020 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:55.114202 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.114207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.114897 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.114898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.115721 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.115724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.116107 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.116108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.116216 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.116739 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:55.116747 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:55.118434 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:55.120074 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:55.120081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:55.120083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:55.120297 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:55.120508 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:55.120509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:55.120510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:55.120618 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:55.120878 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:55.120884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:55.121018 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 07:20:55.121121 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 07:20:55.121277 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260924 07:20:55.122092 18133 heartbeater.cc:347] Connected to a master server at 127.3.246.126:43111
I20260924 07:20:55.122181 18133 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:55.122367 18133 heartbeater.cc:511] Master 127.3.246.126:43111 requested a full tablet report, sending...
I20260924 07:20:55.122723 17808 ts_manager.cc:194] Registered new tserver with Master: e24f48a3e33449ed9d349aa731765fc4 (127.3.246.66:35815)
I20260924 07:20:55.123262 17808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:44363
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:20:55.144614 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:43111
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:55.223021 18138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:55.223202 18138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:55.223224 18138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:55.224298 18138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:55.224339 18138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:55.224364 18138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:20:55.225703 18138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:55.225870 18138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:55.226022 18138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:55.230623 18144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:55.230633 18146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:55.230656 18143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:55.230962 18138 server_base.cc:1034] running on GCE node
I20260924 07:20:55.231163 18138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:55.231411 18138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:55.232574 18138 hybrid_clock.cc:648] HybridClock initialized: now 1790234455232561 us; error 29 us; skew 500 ppm
Sep 24 07:20:55 dist-test-slave-rc1w krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:55.235061 18138 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:20:55.235566 18138 webserver.cc:460] Webserver started at http://127.3.246.67:39971/ using document root <none> and password file <none>
I20260924 07:20:55.235812 18138 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:55.235880 18138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:55.236032 18138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:55.237111 18138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "da63ac1c49c741448c869fd8bbe50544"
format_stamp: "Formatted at 2026-09-24 07:20:55 on dist-test-slave-rc1w"
server_key: "77b8c8b23a156c5a130da222e59ff850"
server_key_iv: "22995d7f0351a35b2732f5c6d351cb91"
server_key_version: "encryptionkey@0"
I20260924 07:20:55.237514 18138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "da63ac1c49c741448c869fd8bbe50544"
format_stamp: "Formatted at 2026-09-24 07:20:55 on dist-test-slave-rc1w"
server_key: "77b8c8b23a156c5a130da222e59ff850"
server_key_iv: "22995d7f0351a35b2732f5c6d351cb91"
server_key_version: "encryptionkey@0"
I20260924 07:20:55.238961 18138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:20:55.239679 18153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.239835 18138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.239931 18138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "da63ac1c49c741448c869fd8bbe50544"
format_stamp: "Formatted at 2026-09-24 07:20:55 on dist-test-slave-rc1w"
server_key: "77b8c8b23a156c5a130da222e59ff850"
server_key_iv: "22995d7f0351a35b2732f5c6d351cb91"
server_key_version: "encryptionkey@0"
I20260924 07:20:55.240000 18138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:55.268302 18138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:55.270328 18138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:55.270488 18138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:55.270699 18138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:55.271009 18138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:55.271059 18138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.271100 18138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:55.271127 18138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.276832 18138 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:34499
I20260924 07:20:55.276933 18266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:34499 every 8 connection(s)
I20260924 07:20:55.277210 18138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:20:55.279127 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 18138
I20260924 07:20:55.279247 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:20:55.279425 4057 external_mini_cluster.cc:1428] Setting key 5d92e298103f467039278808cfb5d27a
Sep 24 07:20:55 dist-test-slave-rc1w krb5kdc[17763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:55.285912 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.277905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:52973 (local address 127.3.246.126:43111)
0924 07:20:55.277986 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.277988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.278154 (+ 166us) server_negotiation.cc:407] Connection header received
0924 07:20:55.278617 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.278618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.278628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.278679 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:55.279541 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.280242 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.281189 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.281398 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.281468 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.283449 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:55.283450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:55.283450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:55.283457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:55.284620 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.285086 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.285087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.285087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.285144 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.285359 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.285359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.285360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.285459 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:55.285512 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.285766 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.285850 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:20:55.286142 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.277881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:52973)
0924 07:20:55.278047 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.278080 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:55.278539 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:55.278727 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:55.278730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:55.279259 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:55.279444 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.279449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.280312 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.280313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.281094 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.281096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.281474 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.281475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.281602 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.282056 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:55.282064 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:55.283370 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:55.284685 (+ 1315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:55.284691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:55.284692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:55.285006 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:55.285205 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:55.285206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:55.285206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:55.285304 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:55.285512 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:55.285516 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:55.285618 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 07:20:55.285758 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 07:20:55.285896 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260924 07:20:55.286703 18267 heartbeater.cc:347] Connected to a master server at 127.3.246.126:43111
I20260924 07:20:55.286794 18267 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:55.286926 18267 heartbeater.cc:511] Master 127.3.246.126:43111 requested a full tablet report, sending...
I20260924 07:20:55.287258 17808 ts_manager.cc:194] Registered new tserver with Master: da63ac1c49c741448c869fd8bbe50544 (127.3.246.67:34499)
I20260924 07:20:55.287606 17808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:52973
I20260924 07:20:55.291093 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:20:55.298271 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.292430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43592 (local address 127.3.246.126:43111)
0924 07:20:55.292485 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.292487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.292587 (+ 100us) server_negotiation.cc:407] Connection header received
0924 07:20:55.292662 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.292663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.292672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.292717 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:55.293315 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.293888 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.294573 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.294754 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.294804 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.295846 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:55.295847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:55.295848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:55.295857 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:55.297247 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.297552 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.297553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.297554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.297602 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.297807 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.297808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.297808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.297942 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:55.298025 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.298110 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.298196 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 07:20:55.299742 17808 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:20:55.301333 17808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:20:55.307479 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.303530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.3.246.66:35815)
0924 07:20:55.304144 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.304149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.304158 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:20:55.304195 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.304197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.304211 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.304282 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:55.304491 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.305159 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.306574 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.307118 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.307177 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.307214 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.307380 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.307422 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":73,"threads_started":1}
I20260924 07:20:55.307495 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.303400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:35815 (local address 127.0.0.1:57188)
0924 07:20:55.303701 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.303739 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:55.303807 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:55.304355 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:55.304355 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:55.304356 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:55.304406 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.304407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.305255 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.305256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.306488 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.306490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.307199 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.307200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.307277 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.307289 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 07:20:55.307361 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:20:55.307413 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":60,"threads_started":1}
I20260924 07:20:55.308127 18067 tablet_service.cc:1467] Processing CreateTablet for tablet 87f18b6552b84491a9c2ecb1ffa15f85 (DEFAULT_TABLE table=test-table [id=fc939c5b1b3e44429d40c0f4fbfc1b0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:55.308341 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.303402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:34499 (local address 127.0.0.1:32880)
0924 07:20:55.303907 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.303915 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:55.303965 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:55.304219 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:55.304222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:55.304224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:55.304280 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.304282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.305526 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.305527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.306608 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.306609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.307341 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.307342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.307405 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.307408 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:20:55.307455 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 07:20:55.307487 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260924 07:20:55.308471 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.303410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:46021 (local address 127.0.0.1:41240)
0924 07:20:55.304024 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.304032 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:55.304073 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:55.304352 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:55.304353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:55.304355 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:55.304411 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.304413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.305608 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.305608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.306597 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.306600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.308221 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.308223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.308280 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.308285 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:20:55.308342 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 07:20:55.308379 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":9,"thread_start_us":53,"threads_started":1}
I20260924 07:20:55.308490 18067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f18b6552b84491a9c2ecb1ffa15f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:55.308594 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.303628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.3.246.67:34499)
0924 07:20:55.303772 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.303776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.303976 (+ 200us) server_negotiation.cc:407] Connection header received
0924 07:20:55.304039 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.304041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.304057 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.304130 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:55.304339 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.305425 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.306681 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.307186 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.307254 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.307309 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.308500 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.308541 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260924 07:20:55.308681 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.303804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.3.246.65:46021)
0924 07:20:55.304003 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.304007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.304131 (+ 124us) server_negotiation.cc:407] Connection header received
0924 07:20:55.304180 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.304182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.304197 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.304268 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:55.304483 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.305520 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.307532 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.308122 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.308181 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.308245 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.308564 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.308609 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":101,"threads_started":1}
I20260924 07:20:55.309127 18201 tablet_service.cc:1467] Processing CreateTablet for tablet 87f18b6552b84491a9c2ecb1ffa15f85 (DEFAULT_TABLE table=test-table [id=fc939c5b1b3e44429d40c0f4fbfc1b0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:55.309345 18201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f18b6552b84491a9c2ecb1ffa15f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:55.310292 17933 tablet_service.cc:1467] Processing CreateTablet for tablet 87f18b6552b84491a9c2ecb1ffa15f85 (DEFAULT_TABLE table=test-table [id=fc939c5b1b3e44429d40c0f4fbfc1b0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:55.310611 17933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f18b6552b84491a9c2ecb1ffa15f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:55.311472 18281 tablet_bootstrap.cc:492] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4: Bootstrap starting.
I20260924 07:20:55.312175 18281 tablet_bootstrap.cc:654] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:55.312404 18282 tablet_bootstrap.cc:492] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544: Bootstrap starting.
I20260924 07:20:55.312822 18281 log.cc:824] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:55.313028 18282 tablet_bootstrap.cc:654] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:55.313535 18281 tablet_bootstrap.cc:492] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4: No bootstrap required, opened a new log
I20260924 07:20:55.313547 18282 log.cc:824] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:55.313629 18281 ts_tablet_manager.cc:1397] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:55.313822 18284 tablet_bootstrap.cc:492] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Bootstrap starting.
I20260924 07:20:55.314253 18282 tablet_bootstrap.cc:492] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544: No bootstrap required, opened a new log
I20260924 07:20:55.314334 18282 ts_tablet_manager.cc:1397] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:20:55.314436 18284 tablet_bootstrap.cc:654] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:55.315001 18284 log.cc:824] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:55.315748 18284 tablet_bootstrap.cc:492] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: No bootstrap required, opened a new log
I20260924 07:20:55.315817 18284 ts_tablet_manager.cc:1397] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:20:55.315881 18281 raft_consensus.cc:348] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.316066 18281 raft_consensus.cc:374] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:55.316102 18281 raft_consensus.cc:729] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e24f48a3e33449ed9d349aa731765fc4, State: Initialized, Role: FOLLOWER
I20260924 07:20:55.316082 18282 raft_consensus.cc:348] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.316236 18281 consensus_queue.cc:260] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.316269 18282 raft_consensus.cc:374] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:55.316310 18282 raft_consensus.cc:729] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da63ac1c49c741448c869fd8bbe50544, State: Initialized, Role: FOLLOWER
I20260924 07:20:55.316434 18282 consensus_queue.cc:260] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.316514 18133 heartbeater.cc:503] Master 127.3.246.126:43111 was elected leader, sending a full tablet report...
I20260924 07:20:55.316715 18267 heartbeater.cc:503] Master 127.3.246.126:43111 was elected leader, sending a full tablet report...
I20260924 07:20:55.316720 18282 ts_tablet_manager.cc:1428] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:55.317132 18281 ts_tablet_manager.cc:1428] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:20:55.317759 18284 raft_consensus.cc:348] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.317896 18284 raft_consensus.cc:374] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:55.317919 18284 raft_consensus.cc:729] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651bd68fbf3d496c9dc50e8af8461e66, State: Initialized, Role: FOLLOWER
I20260924 07:20:55.318011 18284 consensus_queue.cc:260] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.318217 18284 ts_tablet_manager.cc:1428] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:55.318270 17999 heartbeater.cc:503] Master 127.3.246.126:43111 was elected leader, sending a full tablet report...
W20260924 07:20:55.362437 18134 tablet.cc:2369] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:20:55.382202 18288 raft_consensus.cc:482] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:55.382320 18288 raft_consensus.cc:504] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.382740 18288 leader_election.cc:288] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e24f48a3e33449ed9d349aa731765fc4 (127.3.246.66:35815), 651bd68fbf3d496c9dc50e8af8461e66 (127.3.246.65:46021)
I20260924 07:20:55.385586 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.382968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:35733 (local address 127.3.246.65:46021)
0924 07:20:55.383026 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.383029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.383038 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:20:55.383071 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.383072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.383081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.383161 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:55.383319 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.383820 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.384728 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.385245 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.385324 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.385332 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.385481 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.385518 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:20:55.385588 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.382890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:46021 (local address 127.3.246.67:35733)
0924 07:20:55.382956 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.382968 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:55.383017 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:55.383165 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:55.383167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:55.383168 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:55.383241 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.383242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.383908 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.383909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.384634 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.384636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.385334 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.385336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.385412 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.385415 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:20:55.385476 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:20:55.385518 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":2}
I20260924 07:20:55.385699 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.382951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:49737 (local address 127.3.246.66:35815)
0924 07:20:55.383020 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.383022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.383227 (+ 205us) server_negotiation.cc:407] Connection header received
0924 07:20:55.383289 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.383290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.383299 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.383339 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:55.383507 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.383958 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.384873 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.385324 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.385406 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.385413 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.385574 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.385611 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:20:55.385900 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.382896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:35815 (local address 127.3.246.67:49737)
0924 07:20:55.383162 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.383169 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:55.383229 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:55.383349 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:55.383350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:55.383351 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:55.383422 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.383423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.384046 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.384047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.384784 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.384786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.385417 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.385418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.385495 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.385506 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 07:20:55.385568 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 07:20:55.385606 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":98,"thread_start_us":110,"threads_started":1}
I20260924 07:20:55.385949 17953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f18b6552b84491a9c2ecb1ffa15f85" candidate_uuid: "da63ac1c49c741448c869fd8bbe50544" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "651bd68fbf3d496c9dc50e8af8461e66" is_pre_election: true
I20260924 07:20:55.386085 17953 raft_consensus.cc:2454] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da63ac1c49c741448c869fd8bbe50544 in term 0.
I20260924 07:20:55.386258 18087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f18b6552b84491a9c2ecb1ffa15f85" candidate_uuid: "da63ac1c49c741448c869fd8bbe50544" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24f48a3e33449ed9d349aa731765fc4" is_pre_election: true
I20260924 07:20:55.386335 18157 leader_election.cc:302] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 651bd68fbf3d496c9dc50e8af8461e66, da63ac1c49c741448c869fd8bbe50544; no voters:
I20260924 07:20:55.386410 18087 raft_consensus.cc:2454] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da63ac1c49c741448c869fd8bbe50544 in term 0.
I20260924 07:20:55.386466 18288 raft_consensus.cc:2781] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:20:55.386512 18288 raft_consensus.cc:482] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:20:55.386533 18288 raft_consensus.cc:3037] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:55.387236 18288 raft_consensus.cc:504] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.387363 18288 leader_election.cc:288] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [CANDIDATE]: Term 1 election: Requested vote from peers e24f48a3e33449ed9d349aa731765fc4 (127.3.246.66:35815), 651bd68fbf3d496c9dc50e8af8461e66 (127.3.246.65:46021)
I20260924 07:20:55.387492 18087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f18b6552b84491a9c2ecb1ffa15f85" candidate_uuid: "da63ac1c49c741448c869fd8bbe50544" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24f48a3e33449ed9d349aa731765fc4"
I20260924 07:20:55.387531 17953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f18b6552b84491a9c2ecb1ffa15f85" candidate_uuid: "da63ac1c49c741448c869fd8bbe50544" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "651bd68fbf3d496c9dc50e8af8461e66"
I20260924 07:20:55.387569 18087 raft_consensus.cc:3037] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:55.387615 17953 raft_consensus.cc:3037] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:55.388310 17953 raft_consensus.cc:2454] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da63ac1c49c741448c869fd8bbe50544 in term 1.
I20260924 07:20:55.388334 18087 raft_consensus.cc:2454] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da63ac1c49c741448c869fd8bbe50544 in term 1.
I20260924 07:20:55.388474 18157 leader_election.cc:302] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 651bd68fbf3d496c9dc50e8af8461e66, da63ac1c49c741448c869fd8bbe50544; no voters:
I20260924 07:20:55.388579 18288 raft_consensus.cc:2781] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:55.388716 18288 raft_consensus.cc:686] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [term 1 LEADER]: Becoming Leader. State: Replica: da63ac1c49c741448c869fd8bbe50544, State: Running, Role: LEADER
I20260924 07:20:55.388801 18288 consensus_queue.cc:237] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.389585 17808 catalog_manager.cc:5495] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 reported cstate change: term changed from 0 to 1, leader changed from <none> to da63ac1c49c741448c869fd8bbe50544 (127.3.246.67). New cstate: current_term: 1 leader_uuid: "da63ac1c49c741448c869fd8bbe50544" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } health_report { overall_health: UNKNOWN } } }
I20260924 07:20:55.397511 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.395389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.3.246.67:34499)
0924 07:20:55.395442 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.395444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.395453 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:20:55.395499 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.395500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.395515 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.395586 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:55.395674 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.396117 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.396861 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.397037 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.397095 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.397199 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:55.397296 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:55.397404 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.397428 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.397445 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 07:20:55.399230 17953 raft_consensus.cc:1261] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 FOLLOWER]: Refusing update from remote peer da63ac1c49c741448c869fd8bbe50544: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:55.399257 18087 raft_consensus.cc:1261] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Refusing update from remote peer da63ac1c49c741448c869fd8bbe50544: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:55.399590 18288 consensus_queue.cc:1035] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:55.399753 18291 consensus_queue.cc:1035] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [LEADER]: Connected to new peer: Peer: permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:55.401877 18294 mvcc.cc:205] Tried to move back new op lower bound from 7332800329313124352 to 7332800329273290752. Current Snapshot: MvccSnapshot[applied={T|T < 7332800329313124352}]
I20260924 07:20:55.402040 18296 mvcc.cc:205] Tried to move back new op lower bound from 7332800329313124352 to 7332800329273290752. Current Snapshot: MvccSnapshot[applied={T|T < 7332800329313124352}]
I20260924 07:20:55.402908 18295 mvcc.cc:205] Tried to move back new op lower bound from 7332800329313124352 to 7332800329273290752. Current Snapshot: MvccSnapshot[applied={T|T < 7332800329313124352 or (T in {7332800329313124352})}]
I20260924 07:20:55.404071 17806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:55.404166 17806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:55.404881 17806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Sending DeleteTablet for 3 replicas of tablet 87f18b6552b84491a9c2ecb1ffa15f85
I20260924 07:20:55.405230 17933 tablet_service.cc:1514] Processing DeleteTablet for tablet 87f18b6552b84491a9c2ecb1ffa15f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:55 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:41240
I20260924 07:20:55.405224 18201 tablet_service.cc:1514] Processing DeleteTablet for tablet 87f18b6552b84491a9c2ecb1ffa15f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:55 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:32880
I20260924 07:20:55.405269 18067 tablet_service.cc:1514] Processing DeleteTablet for tablet 87f18b6552b84491a9c2ecb1ffa15f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:55 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:57188
I20260924 07:20:55.405483 18306 tablet_replica.cc:330] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4: stopping tablet replica
I20260924 07:20:55.405476 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 17870
I20260924 07:20:55.405575 18306 raft_consensus.cc:2229] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:55.406122 18306 raft_consensus.cc:2258] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:55.406951 18306 ts_tablet_manager.cc:1905] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:55.407658 18304 tablet_replica.cc:330] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544: stopping tablet replica
I20260924 07:20:55.407795 18304 raft_consensus.cc:2229] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:20:55.407974 18304 raft_consensus.cc:2258] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:55.408468 18304 ts_tablet_manager.cc:1905] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:55.408831 18306 ts_tablet_manager.cc:1918] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:55.408910 18306 log.cc:1196] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/87f18b6552b84491a9c2ecb1ffa15f85
I20260924 07:20:55.409185 18306 ts_tablet_manager.cc:1939] T 87f18b6552b84491a9c2ecb1ffa15f85 P e24f48a3e33449ed9d349aa731765fc4: Deleting consensus metadata
I20260924 07:20:55.409574 17792 catalog_manager.cc:4841] TS e24f48a3e33449ed9d349aa731765fc4 (127.3.246.66:35815): tablet 87f18b6552b84491a9c2ecb1ffa15f85 (table test-table [id=fc939c5b1b3e44429d40c0f4fbfc1b0c]) successfully deleted
I20260924 07:20:55.410203 18304 ts_tablet_manager.cc:1918] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:55.410283 18304 log.cc:1196] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/87f18b6552b84491a9c2ecb1ffa15f85
I20260924 07:20:55.410552 18304 ts_tablet_manager.cc:1939] T 87f18b6552b84491a9c2ecb1ffa15f85 P da63ac1c49c741448c869fd8bbe50544: Deleting consensus metadata
I20260924 07:20:55.410943 17794 catalog_manager.cc:4841] TS da63ac1c49c741448c869fd8bbe50544 (127.3.246.67:34499): tablet 87f18b6552b84491a9c2ecb1ffa15f85 (table test-table [id=fc939c5b1b3e44429d40c0f4fbfc1b0c]) successfully deleted
W20260924 07:20:55.411926 17795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:46021 (error 108)
I20260924 07:20:55.412418 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.412283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:46021 (local address 127.0.0.1:41246)
0924 07:20:55.412339 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.412383 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 07:20:55.412478 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 18004
W20260924 07:20:55.412536 17795 catalog_manager.cc:4584] TS 651bd68fbf3d496c9dc50e8af8461e66 (127.3.246.65:46021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87f18b6552b84491a9c2ecb1ffa15f85: Network error: Client connection negotiation failed: client connection to 127.3.246.65:46021: connect: Connection refused (error 111)
I20260924 07:20:55.418859 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 18138
I20260924 07:20:55.424031 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 17776
I20260924 07:20:55.429171 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:43111
--webserver_interface=127.3.246.126
--webserver_port=45081
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:43111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:55.509150 18308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:55.509351 18308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:55.509372 18308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:55.510462 18308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:55.510497 18308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:55.510511 18308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:55.510524 18308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:55.510535 18308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:55.511898 18308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:43111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:43111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=45081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:55.512075 18308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:55.512236 18308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:55.520431 18314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:55.520496 18317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:55.520507 18315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:55.520642 18308 server_base.cc:1034] running on GCE node
I20260924 07:20:55.520915 18308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:55.521198 18308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:55.522342 18308 hybrid_clock.cc:648] HybridClock initialized: now 1790234455522326 us; error 25 us; skew 500 ppm
Sep 24 07:20:55 dist-test-slave-rc1w krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:55.524874 18308 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:20:55.525331 18308 webserver.cc:460] Webserver started at http://127.3.246.126:45081/ using document root <none> and password file <none>
I20260924 07:20:55.525571 18308 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:55.525650 18308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:55.527078 18308 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:20:55.527729 18324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.527885 18308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:20:55.527983 18308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "1305b1d1f3a74376abb0278f7860a18d"
format_stamp: "Formatted at 2026-09-24 07:20:54 on dist-test-slave-rc1w"
server_key: "ad40f7e39b6abc2d78f6754a43584563"
server_key_iv: "c88529992ed1c161988c328f9d2dd9e0"
server_key_version: "encryptionkey@0"
I20260924 07:20:55.528353 18308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:55.545554 18308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:55.547824 18308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:55.547977 18308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:55.551991 18308 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:43111
I20260924 07:20:55.552038 18376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:43111 every 8 connection(s)
I20260924 07:20:55.552423 18308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:55.553771 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 18308
I20260924 07:20:55.554090 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:46021
--local_ip_for_outbound_sockets=127.3.246.65
--tserver_master_addrs=127.3.246.126:43111
--webserver_port=36929
--webserver_interface=127.3.246.65
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:20:55.554170 18377 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:20:55.555229 18377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Bootstrap starting.
I20260924 07:20:55.563561 18377 log.cc:824] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:55.565472 18377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:20:55.565812 18377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Bootstrap complete.
I20260924 07:20:55.568212 18377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } }
I20260924 07:20:55.568431 18377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1305b1d1f3a74376abb0278f7860a18d, State: Initialized, Role: FOLLOWER
I20260924 07:20:55.568594 18377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } }
I20260924 07:20:55.568653 18377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:55.568717 18377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:55.568766 18377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:20:55.569818 18377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } }
I20260924 07:20:55.569900 18377 leader_election.cc:302] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1305b1d1f3a74376abb0278f7860a18d; no voters:
I20260924 07:20:55.570107 18377 leader_election.cc:288] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:20:55.570140 18381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:20:55.570252 18381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 2 LEADER]: Becoming Leader. State: Replica: 1305b1d1f3a74376abb0278f7860a18d, State: Running, Role: LEADER
I20260924 07:20:55.570343 18381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } }
I20260924 07:20:55.570375 18377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:55.570606 18381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1305b1d1f3a74376abb0278f7860a18d. Latest consensus state: current_term: 2 leader_uuid: "1305b1d1f3a74376abb0278f7860a18d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } } }
I20260924 07:20:55.570683 18381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:55.570611 18382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1305b1d1f3a74376abb0278f7860a18d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } } }
I20260924 07:20:55.570853 18382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:55.570869 18385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:55.571717 18385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:55.571826 18385 catalog_manager.cc:1259] Loaded cluster ID: 22e3036aea534bcda916c64eb1afc287
I20260924 07:20:55.571866 18385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:55.572152 18385 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 07:20:55.574877 18385 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 07:20:55.576665 18385 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:55.576871 18385 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Loaded TSK: 0
I20260924 07:20:55.577090 18385 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:20:55.645035 18379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:55.645232 18379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:55.645255 18379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:55.646478 18379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:55.646517 18379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:55.646543 18379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:20:55.647957 18379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:46021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=36929
--tserver_master_addrs=127.3.246.126:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:55.648181 18379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:55.648423 18379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:55.653160 18405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:55.653172 18407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:55.653255 18404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:55.654269 18379 server_base.cc:1034] running on GCE node
I20260924 07:20:55.654505 18379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:55.654778 18379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:55.655983 18379 hybrid_clock.cc:648] HybridClock initialized: now 1790234455655952 us; error 29 us; skew 500 ppm
Sep 24 07:20:55 dist-test-slave-rc1w krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:55.658936 18379 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:20:55.659406 18379 webserver.cc:460] Webserver started at http://127.3.246.65:36929/ using document root <none> and password file <none>
I20260924 07:20:55.659628 18379 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:55.659708 18379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:55.661092 18379 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:55.661849 18414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.662062 18379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:55.662164 18379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "651bd68fbf3d496c9dc50e8af8461e66"
format_stamp: "Formatted at 2026-09-24 07:20:54 on dist-test-slave-rc1w"
server_key: "bc3788d2c75c8a70e639680238efc587"
server_key_iv: "a553ea6bd34c46427b73b130c75a0b2f"
server_key_version: "encryptionkey@0"
I20260924 07:20:55.662518 18379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:55.670749 18379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:55.672932 18379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:55.673076 18379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:55.673277 18379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:55.673712 18421 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:20:55.675565 18379 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:20:55.675617 18379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 07:20:55.675659 18379 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:20:55.676254 18379 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:20:55.676304 18379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 07:20:55.676373 18421 tablet_bootstrap.cc:492] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Bootstrap starting.
I20260924 07:20:55.683632 18379 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:46021
I20260924 07:20:55.684005 18379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:20:55.687000 18528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:46021 every 8 connection(s)
I20260924 07:20:55.687330 18421 log.cc:824] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:55.689167 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 18379
I20260924 07:20:55.689476 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:35815
--local_ip_for_outbound_sockets=127.3.246.66
--tserver_master_addrs=127.3.246.126:43111
--webserver_port=35095
--webserver_interface=127.3.246.66
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:20:55.690007 18421 tablet_bootstrap.cc:492] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:20:55.690300 18421 tablet_bootstrap.cc:492] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Bootstrap complete.
I20260924 07:20:55.691514 18421 ts_tablet_manager.cc:1397] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260924 07:20:55.693298 18421 raft_consensus.cc:348] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.693650 18421 raft_consensus.cc:729] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651bd68fbf3d496c9dc50e8af8461e66, State: Initialized, Role: FOLLOWER
I20260924 07:20:55.693814 18421 consensus_queue.cc:260] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.694072 18421 ts_tablet_manager.cc:1428] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
Sep 24 07:20:55 dist-test-slave-rc1w krb5kdc[17763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:55.698063 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.684631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:35703)
0924 07:20:55.685794 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.685832 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:55.686261 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:55.686771 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:55.686775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:55.687230 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:55.687443 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.687449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.689452 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.689453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.691156 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.691159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.692156 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.692157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.692298 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.692901 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:55.692911 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:55.694680 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:55.696574 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:55.696581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:55.696583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:55.696802 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:55.697050 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:55.697051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:55.697051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:55.697158 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:55.697513 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:55.697519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:55.697648 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 07:20:55.697781 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 07:20:55.697929 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"thread_start_us":105,"threads_started":1}
I20260924 07:20:55.698638 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.684771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:35703 (local address 127.3.246.126:43111)
0924 07:20:55.686426 (+ 1655us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.686434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.686460 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:20:55.686568 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.686572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.686610 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.686743 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:55.687547 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.689316 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.691251 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.691946 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.692050 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.694802 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:55.694807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:55.694807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:55.694818 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:55.696472 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.696887 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.696890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.696890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.696982 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.697228 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.697228 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.697229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.697437 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:55.697521 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.698354 (+ 833us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.698493 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1602,"thread_start_us":67,"threads_started":1}
I20260924 07:20:55.699281 18529 heartbeater.cc:347] Connected to a master server at 127.3.246.126:43111
I20260924 07:20:55.699385 18529 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:55.699563 18529 heartbeater.cc:511] Master 127.3.246.126:43111 requested a full tablet report, sending...
I20260924 07:20:55.700215 18341 ts_manager.cc:194] Registered new tserver with Master: 651bd68fbf3d496c9dc50e8af8461e66 (127.3.246.65:46021)
I20260924 07:20:55.701279 18341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:35703
I20260924 07:20:55.702072 18529 heartbeater.cc:503] Master 127.3.246.126:43111 was elected leader, sending a full tablet report...
I20260924 07:20:55.708709 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.700992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.3.246.65:46021)
0924 07:20:55.701145 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.701149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.705869 (+ 4720us) server_negotiation.cc:407] Connection header received
0924 07:20:55.705969 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.705971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.705986 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.706072 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:55.706232 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.706866 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.707847 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.708291 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.708371 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.708436 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.708608 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.708644 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260924 07:20:55.708719 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.700751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:46021 (local address 127.0.0.1:41262)
0924 07:20:55.705778 (+ 5027us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.705807 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:55.705885 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:55.706076 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:55.706078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:55.706080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:55.706158 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.706160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.706966 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.706968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.707761 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.707764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.708388 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.708390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.708487 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.708499 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 07:20:55.708591 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 07:20:55.708644 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4995,"thread_start_us":85,"threads_started":1}
I20260924 07:20:55.709318 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet 87f18b6552b84491a9c2ecb1ffa15f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:55 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:41262
I20260924 07:20:55.709852 18538 tablet_replica.cc:330] stopping tablet replica
I20260924 07:20:55.709981 18538 raft_consensus.cc:2229] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:55.710047 18538 raft_consensus.cc:2258] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:55.710342 18538 ts_tablet_manager.cc:1905] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:55.712314 18538 ts_tablet_manager.cc:1918] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:55.712394 18538 log.cc:1196] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal/wals/87f18b6552b84491a9c2ecb1ffa15f85
I20260924 07:20:55.712625 18538 ts_tablet_manager.cc:1939] T 87f18b6552b84491a9c2ecb1ffa15f85 P 651bd68fbf3d496c9dc50e8af8461e66: Deleting consensus metadata
I20260924 07:20:55.713143 18328 catalog_manager.cc:4841] TS 651bd68fbf3d496c9dc50e8af8461e66 (127.3.246.65:46021): tablet 87f18b6552b84491a9c2ecb1ffa15f85 (table test-table [id=fc939c5b1b3e44429d40c0f4fbfc1b0c]) successfully deleted
W20260924 07:20:55.781503 18534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:55.781744 18534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:55.781795 18534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:55.782931 18534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:55.782970 18534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:55.782996 18534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:20:55.785007 18534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:35815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=35095
--tserver_master_addrs=127.3.246.126:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:55.785264 18534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:55.785497 18534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:55.790624 18546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:55.790630 18534 server_base.cc:1034] running on GCE node
W20260924 07:20:55.790616 18544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:55.790613 18543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:55.791030 18534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:55.791283 18534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:55.792438 18534 hybrid_clock.cc:648] HybridClock initialized: now 1790234455792422 us; error 26 us; skew 500 ppm
Sep 24 07:20:55 dist-test-slave-rc1w krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:55.794962 18534 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:20:55.795356 18534 webserver.cc:460] Webserver started at http://127.3.246.66:35095/ using document root <none> and password file <none>
I20260924 07:20:55.795584 18534 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:55.795663 18534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:55.796926 18534 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:55.797433 18553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.797598 18534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:20:55.797708 18534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "e24f48a3e33449ed9d349aa731765fc4"
format_stamp: "Formatted at 2026-09-24 07:20:55 on dist-test-slave-rc1w"
server_key: "03d2f076bee418ce3e2bc20f2abc0481"
server_key_iv: "54868eb8b1d9ae3fe489a876a8a9be81"
server_key_version: "encryptionkey@0"
I20260924 07:20:55.798074 18534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:55.811066 18534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:55.812819 18534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:55.812973 18534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:55.813185 18534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:55.813472 18534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:55.813525 18534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.813566 18534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:55.813591 18534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.819255 18534 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:35815
I20260924 07:20:55.819303 18666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:35815 every 8 connection(s)
I20260924 07:20:55.819610 18534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:20:55.825477 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 18534
I20260924 07:20:55.825805 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:34499
--local_ip_for_outbound_sockets=127.3.246.67
--tserver_master_addrs=127.3.246.126:43111
--webserver_port=39971
--webserver_interface=127.3.246.67
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:20:55 dist-test-slave-rc1w krb5kdc[17763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:55.829450 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.820295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:54009 (local address 127.3.246.126:43111)
0924 07:20:55.820369 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.820371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.820572 (+ 201us) server_negotiation.cc:407] Connection header received
0924 07:20:55.821090 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.821091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.821107 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.821164 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:55.822170 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.822671 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.823662 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.823877 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.823973 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.826367 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:55.826369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:55.826369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:55.826378 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:55.827855 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.828284 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.828285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.828286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.828350 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.828671 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.828672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.828672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.828835 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:55.828902 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.829229 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.829330 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:20:55.829548 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.820286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:54009)
0924 07:20:55.820462 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.820496 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:55.820989 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:55.821230 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:55.821233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:55.821866 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:55.822056 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.822061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.822770 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.822771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.823555 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.823557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.823988 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.823989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.824109 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.824659 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:55.824666 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:55.826270 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:55.827934 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:55.827941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:55.827943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:55.828197 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:55.828442 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:55.828443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:55.828443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:55.828607 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:55.828892 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:55.828899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:55.829068 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 07:20:55.829193 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 07:20:55.829369 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260924 07:20:55.830030 18667 heartbeater.cc:347] Connected to a master server at 127.3.246.126:43111
I20260924 07:20:55.830119 18667 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:55.830304 18667 heartbeater.cc:511] Master 127.3.246.126:43111 requested a full tablet report, sending...
I20260924 07:20:55.830696 18341 ts_manager.cc:194] Registered new tserver with Master: e24f48a3e33449ed9d349aa731765fc4 (127.3.246.66:35815)
I20260924 07:20:55.831246 18341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:54009
W20260924 07:20:55.907124 18670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:55.907342 18670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:55.907382 18670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:55.908511 18670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:55.908576 18670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:55.908622 18670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:20:55.910104 18670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:34499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=39971
--tserver_master_addrs=127.3.246.126:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:55.910365 18670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:55.910578 18670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:55.915462 18676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:55.915525 18670 server_base.cc:1034] running on GCE node
W20260924 07:20:55.915499 18675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:55.915639 18678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:55.915834 18670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:55.916080 18670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:55.917217 18670 hybrid_clock.cc:648] HybridClock initialized: now 1790234455917193 us; error 47 us; skew 500 ppm
Sep 24 07:20:55 dist-test-slave-rc1w krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:55.919591 18670 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:20:55.920010 18670 webserver.cc:460] Webserver started at http://127.3.246.67:39971/ using document root <none> and password file <none>
I20260924 07:20:55.920183 18670 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:55.920233 18670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:55.921511 18670 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:55.922067 18685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.922226 18670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.922307 18670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "da63ac1c49c741448c869fd8bbe50544"
format_stamp: "Formatted at 2026-09-24 07:20:55 on dist-test-slave-rc1w"
server_key: "77b8c8b23a156c5a130da222e59ff850"
server_key_iv: "22995d7f0351a35b2732f5c6d351cb91"
server_key_version: "encryptionkey@0"
I20260924 07:20:55.922621 18670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:55.931015 18670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:55.932977 18670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:55.933120 18670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:55.933331 18670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:55.933609 18670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:55.933640 18670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.933660 18670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:55.933722 18670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.939450 18670 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:34499
I20260924 07:20:55.939491 18798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:34499 every 8 connection(s)
I20260924 07:20:55.939805 18670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:20:55.940863 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 18670
Sep 24 07:20:55 dist-test-slave-rc1w krb5kdc[17763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:20:55.947683 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.941212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.3.246.126:43111)
0924 07:20:55.941306 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.941308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.941381 (+ 73us) server_negotiation.cc:407] Connection header received
0924 07:20:55.941470 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.941471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.941482 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.941536 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:55.942149 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.942649 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.943364 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.943576 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.943656 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.944940 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:55.944941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:55.944942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:55.944948 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:55.946556 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.946902 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.946903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.946903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.946960 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.947148 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.947148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.947149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.947269 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:55.947465 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.947502 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.947579 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":49,"threads_started":1}
I20260924 07:20:55.948836 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.940448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:37483 (local address 127.3.246.126:43111)
0924 07:20:55.940506 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.940508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.940724 (+ 216us) server_negotiation.cc:407] Connection header received
0924 07:20:55.941325 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.941326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.941337 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.941392 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:55.942371 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.942831 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.943812 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.944021 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.944103 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.946042 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:55.946043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:55.946044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:55.946051 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:55.947439 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.947944 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.947945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.947946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.947999 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:55.948298 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:55.948298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:55.948299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:55.948426 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:55.948479 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.948700 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.948758 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 07:20:55.948961 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.940438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:37483)
0924 07:20:55.940602 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.940635 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:55.941232 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:55.941460 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:55.941472 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:55.942092 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:55.942270 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.942275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.942917 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.942918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.943728 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.943730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.944114 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.944115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.944229 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.944678 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:55.944686 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:55.945953 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:55.947536 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:55.947543 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:55.947545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:55.947859 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:55.948096 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:55.948097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:55.948097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:55.948232 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:55.948476 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:55.948481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:55.948595 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 07:20:55.948678 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:20:55.948789 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260924 07:20:55.949369 18799 heartbeater.cc:347] Connected to a master server at 127.3.246.126:43111
I20260924 07:20:55.949458 18799 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:55.949623 18799 heartbeater.cc:511] Master 127.3.246.126:43111 requested a full tablet report, sending...
I20260924 07:20:55.949983 18341 ts_manager.cc:194] Registered new tserver with Master: da63ac1c49c741448c869fd8bbe50544 (127.3.246.67:34499)
I20260924 07:20:55.950299 18341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:37483
I20260924 07:20:55.959209 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:20:55.962005 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.959552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.3.246.126:43111)
0924 07:20:55.959598 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.959600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.959695 (+ 95us) server_negotiation.cc:407] Connection header received
0924 07:20:55.959790 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.959791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.959820 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.959903 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:55.960009 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.960453 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.961212 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.961398 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.961472 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.961651 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:55.961780 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:55.961862 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.961915 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.961931 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:20:55.962242 18341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:20:55.963707 18341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:20:55.965941 18451 tablet_service.cc:1467] Processing CreateTablet for tablet 11e97d352bdd405282d360ecfa962a8f (DEFAULT_TABLE table=test-table [id=780f2c4f4b014d3ebc51abd93711a0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:55.966147 18451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e97d352bdd405282d360ecfa962a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:55.967587 18421 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66: Bootstrap starting.
I20260924 07:20:55.967958 18421 tablet_bootstrap.cc:654] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:55.968575 18421 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66: No bootstrap required, opened a new log
I20260924 07:20:55.968617 18421 ts_tablet_manager.cc:1397] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:20:55.968742 18421 raft_consensus.cc:348] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.968804 18421 raft_consensus.cc:374] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:55.968878 18421 raft_consensus.cc:729] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651bd68fbf3d496c9dc50e8af8461e66, State: Initialized, Role: FOLLOWER
I20260924 07:20:55.968932 18421 consensus_queue.cc:260] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.969033 18421 ts_tablet_manager.cc:1428] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:55.969273 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.965472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:35815 (local address 127.0.0.1:57204)
0924 07:20:55.965651 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.965662 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:55.965728 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:55.965969 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:55.965970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:55.965971 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:55.965999 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.966000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.967173 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.967174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.967946 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.967948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.969014 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.969015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.969084 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.969088 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:20:55.969157 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:20:55.969199 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260924 07:20:55.969283 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.965578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.3.246.66:35815)
0924 07:20:55.965793 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.965798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.965809 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:20:55.965877 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.965879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.965894 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.965974 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:55.966046 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.967070 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.968148 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.968903 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.969007 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.969064 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.969173 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.969210 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":136,"threads_started":1}
I20260924 07:20:55.970064 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 11e97d352bdd405282d360ecfa962a8f (DEFAULT_TABLE table=test-table [id=780f2c4f4b014d3ebc51abd93711a0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:55.970357 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e97d352bdd405282d360ecfa962a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:55.971665 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.965520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:34499 (local address 127.0.0.1:32896)
0924 07:20:55.965660 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:55.965701 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:55.965751 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:55.968077 (+ 2326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:55.968078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:55.968080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:55.968116 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.968118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.969620 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.969621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.970578 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:55.970581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.971438 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.971439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:55.971510 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.971514 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:20:55.971561 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 07:20:55.971600 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260924 07:20:55.971693 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:55.967638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.3.246.67:34499)
0924 07:20:55.967836 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:55.967840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:55.967852 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:20:55.967888 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:55.967890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:55.967905 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:55.967994 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:55.968201 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.969536 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.970654 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:55.971313 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:55.971435 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:55.971491 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:55.971588 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 07:20:55.971622 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260924 07:20:55.972397 18733 tablet_service.cc:1467] Processing CreateTablet for tablet 11e97d352bdd405282d360ecfa962a8f (DEFAULT_TABLE table=test-table [id=780f2c4f4b014d3ebc51abd93711a0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:55.972638 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e97d352bdd405282d360ecfa962a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:55.973345 18809 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: Bootstrap starting.
I20260924 07:20:55.974061 18809 tablet_bootstrap.cc:654] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:55.974537 18809 log.cc:824] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:55.975185 18809 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: No bootstrap required, opened a new log
I20260924 07:20:55.975261 18809 ts_tablet_manager.cc:1397] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:20:55.975347 18811 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Bootstrap starting.
I20260924 07:20:55.975868 18811 tablet_bootstrap.cc:654] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:55.976286 18811 log.cc:824] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:55.976872 18809 raft_consensus.cc:348] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.977030 18809 raft_consensus.cc:374] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:55.977077 18809 raft_consensus.cc:729] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e24f48a3e33449ed9d349aa731765fc4, State: Initialized, Role: FOLLOWER
I20260924 07:20:55.977206 18811 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: No bootstrap required, opened a new log
I20260924 07:20:55.977205 18809 consensus_queue.cc:260] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.977274 18811 ts_tablet_manager.cc:1397] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:20:55.977460 18809 ts_tablet_manager.cc:1428] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:20:55.977480 18667 heartbeater.cc:503] Master 127.3.246.126:43111 was elected leader, sending a full tablet report...
I20260924 07:20:55.979015 18811 raft_consensus.cc:348] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.979139 18811 raft_consensus.cc:374] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:55.979161 18811 raft_consensus.cc:729] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da63ac1c49c741448c869fd8bbe50544, State: Initialized, Role: FOLLOWER
I20260924 07:20:55.979283 18811 consensus_queue.cc:260] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:55.979529 18811 ts_tablet_manager.cc:1428] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:55.979549 18799 heartbeater.cc:503] Master 127.3.246.126:43111 was elected leader, sending a full tablet report...
I20260924 07:20:56.049487 18535 raft_consensus.cc:482] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:56.049649 18535 raft_consensus.cc:504] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:56.050055 18535 leader_election.cc:288] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e24f48a3e33449ed9d349aa731765fc4 (127.3.246.66:35815), da63ac1c49c741448c869fd8bbe50544 (127.3.246.67:34499)
I20260924 07:20:56.052798 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.050314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:49161 (local address 127.3.246.67:34499)
0924 07:20:56.050373 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:56.050375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:56.050384 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:20:56.050412 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:56.050413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:56.050422 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:56.050478 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:56.050650 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:56.051136 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:56.052073 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:56.052460 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:56.052541 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:56.052550 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:56.052694 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 07:20:56.052733 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:20:56.052798 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.050226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:34499 (local address 127.3.246.65:49161)
0924 07:20:56.050277 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.050291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:56.050350 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:56.050499 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:56.050499 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:56.050501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:56.050577 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:56.050579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:56.051223 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:56.051224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:56.051976 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:56.051978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:56.052538 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:56.052539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:56.052622 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:56.052625 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:20:56.052687 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 07:20:56.052727 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 07:20:56.053051 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.050337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:44755 (local address 127.3.246.66:35815)
0924 07:20:56.050383 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:56.050385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:56.050527 (+ 142us) server_negotiation.cc:407] Connection header received
0924 07:20:56.050599 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:56.050600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:56.050609 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:56.050672 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:56.050821 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:56.051282 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:56.052203 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:56.052586 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:56.052677 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:56.052685 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:56.052891 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 07:20:56.052939 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:20:56.053190 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e97d352bdd405282d360ecfa962a8f" candidate_uuid: "651bd68fbf3d496c9dc50e8af8461e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da63ac1c49c741448c869fd8bbe50544" is_pre_election: true
I20260924 07:20:56.053309 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.050226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:35815 (local address 127.3.246.65:44755)
0924 07:20:56.050438 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.050452 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:56.050534 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:56.050674 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:56.050675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:56.050677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:56.050753 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:56.050755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:56.051362 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:56.051363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:56.052107 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:56.052108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:56.052681 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:56.052682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:56.052772 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:56.052775 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:20:56.053175 (+ 400us) client_negotiation.cc:240] Negotiation successful
0924 07:20:56.053220 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":52,"thread_start_us":114,"threads_started":1}
I20260924 07:20:56.053340 18753 raft_consensus.cc:2454] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 651bd68fbf3d496c9dc50e8af8461e66 in term 0.
I20260924 07:20:56.053589 18417 leader_election.cc:302] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 651bd68fbf3d496c9dc50e8af8461e66, da63ac1c49c741448c869fd8bbe50544; no voters:
I20260924 07:20:56.053640 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e97d352bdd405282d360ecfa962a8f" candidate_uuid: "651bd68fbf3d496c9dc50e8af8461e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24f48a3e33449ed9d349aa731765fc4" is_pre_election: true
I20260924 07:20:56.053731 18535 raft_consensus.cc:2781] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:20:56.053812 18535 raft_consensus.cc:482] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:20:56.053812 18621 raft_consensus.cc:2454] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 651bd68fbf3d496c9dc50e8af8461e66 in term 0.
I20260924 07:20:56.053848 18535 raft_consensus.cc:3037] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:56.054555 18535 raft_consensus.cc:504] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:56.054699 18535 leader_election.cc:288] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [CANDIDATE]: Term 1 election: Requested vote from peers e24f48a3e33449ed9d349aa731765fc4 (127.3.246.66:35815), da63ac1c49c741448c869fd8bbe50544 (127.3.246.67:34499)
I20260924 07:20:56.054844 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e97d352bdd405282d360ecfa962a8f" candidate_uuid: "651bd68fbf3d496c9dc50e8af8461e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24f48a3e33449ed9d349aa731765fc4"
I20260924 07:20:56.054854 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e97d352bdd405282d360ecfa962a8f" candidate_uuid: "651bd68fbf3d496c9dc50e8af8461e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da63ac1c49c741448c869fd8bbe50544"
I20260924 07:20:56.054934 18621 raft_consensus.cc:3037] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:56.054930 18753 raft_consensus.cc:3037] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:56.055625 18753 raft_consensus.cc:2454] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 651bd68fbf3d496c9dc50e8af8461e66 in term 1.
I20260924 07:20:56.055648 18621 raft_consensus.cc:2454] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 651bd68fbf3d496c9dc50e8af8461e66 in term 1.
I20260924 07:20:56.055820 18415 leader_election.cc:302] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 651bd68fbf3d496c9dc50e8af8461e66, e24f48a3e33449ed9d349aa731765fc4; no voters:
I20260924 07:20:56.055953 18535 raft_consensus.cc:2781] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:56.056097 18535 raft_consensus.cc:686] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 LEADER]: Becoming Leader. State: Replica: 651bd68fbf3d496c9dc50e8af8461e66, State: Running, Role: LEADER
I20260924 07:20:56.056150 18535 consensus_queue.cc:237] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:56.056890 18341 catalog_manager.cc:5495] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 651bd68fbf3d496c9dc50e8af8461e66 (127.3.246.65). New cstate: current_term: 1 leader_uuid: "651bd68fbf3d496c9dc50e8af8461e66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } health_report { overall_health: HEALTHY } } }
W20260924 07:20:56.070132 18668 tablet.cc:2369] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:20:56.084064 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.082058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.3.246.65:46021)
0924 07:20:56.082109 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:56.082111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:56.082120 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:20:56.082162 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:56.082162 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:56.082181 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:56.082246 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:56.082348 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:56.082812 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:56.083465 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:56.083639 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:56.083687 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:56.083807 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:56.083894 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:56.083951 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:56.083991 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 07:20:56.084010 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 07:20:56.085551 18621 raft_consensus.cc:1261] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Refusing update from remote peer 651bd68fbf3d496c9dc50e8af8461e66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:56.085551 18753 raft_consensus.cc:1261] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [term 1 FOLLOWER]: Refusing update from remote peer 651bd68fbf3d496c9dc50e8af8461e66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:56.085923 18535 consensus_queue.cc:1035] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:56.086082 18816 consensus_queue.cc:1035] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:56.091143 18340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:56.091245 18340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:56.091954 18340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Sending DeleteTablet for 3 replicas of tablet 11e97d352bdd405282d360ecfa962a8f
I20260924 07:20:56.092281 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e97d352bdd405282d360ecfa962a8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:56 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:41262
I20260924 07:20:56.092278 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e97d352bdd405282d360ecfa962a8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:56 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:57204
I20260924 07:20:56.092300 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e97d352bdd405282d360ecfa962a8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:56 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:32896
I20260924 07:20:56.092365 18538 tablet_replica.cc:330] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66: stopping tablet replica
I20260924 07:20:56.092432 18538 raft_consensus.cc:2229] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:20:56.092505 18538 raft_consensus.cc:2258] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:56.092743 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 18379
I20260924 07:20:56.092850 18538 ts_tablet_manager.cc:1905] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:20:56.097766 18328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:46021 (error 108)
I20260924 07:20:56.097968 18830 tablet_replica.cc:330] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: stopping tablet replica
I20260924 07:20:56.098080 18830 raft_consensus.cc:2229] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:56.098157 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 18534
I20260924 07:20:56.098183 18830 raft_consensus.cc:2258] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:56.098193 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.098073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:46021 (local address 127.0.0.1:41286)
0924 07:20:56.098116 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.098159 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 07:20:56.098309 18829 tablet_replica.cc:330] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: stopping tablet replica
I20260924 07:20:56.098402 18829 raft_consensus.cc:2229] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:56.098491 18829 raft_consensus.cc:2258] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 07:20:56.099154 18328 catalog_manager.cc:4584] TS 651bd68fbf3d496c9dc50e8af8461e66 (127.3.246.65:46021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11e97d352bdd405282d360ecfa962a8f: Network error: Client connection negotiation failed: client connection to 127.3.246.65:46021: connect: Connection refused (error 111)
I20260924 07:20:56.099766 18829 ts_tablet_manager.cc:1905] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:56.101626 18829 ts_tablet_manager.cc:1918] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:56.101738 18829 log.cc:1196] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/11e97d352bdd405282d360ecfa962a8f
I20260924 07:20:56.102006 18829 ts_tablet_manager.cc:1939] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Deleting consensus metadata
I20260924 07:20:56.102404 18327 catalog_manager.cc:4841] TS da63ac1c49c741448c869fd8bbe50544 (127.3.246.67:34499): tablet 11e97d352bdd405282d360ecfa962a8f (table test-table [id=780f2c4f4b014d3ebc51abd93711a0d7]) successfully deleted
I20260924 07:20:56.102911 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 18670
I20260924 07:20:56.102945 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.102837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:35815 (local address 127.0.0.1:57208)
0924 07:20:56.102882 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.102914 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.66:35815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 07:20:56.107578 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 18308
I20260924 07:20:56.112833 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:43111
--webserver_interface=127.3.246.126
--webserver_port=45081
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:43111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:56.191624 18832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:56.191855 18832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:56.191879 18832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:56.193120 18832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:56.193157 18832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:56.193171 18832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:56.193214 18832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:56.193239 18832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:56.194675 18832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:43111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:43111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=45081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:56.194913 18832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:56.195125 18832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:56.199748 18840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:56.199788 18838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:56.199788 18837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:56.199990 18832 server_base.cc:1034] running on GCE node
I20260924 07:20:56.200199 18832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:56.200471 18832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:56.201642 18832 hybrid_clock.cc:648] HybridClock initialized: now 1790234456201635 us; error 33 us; skew 500 ppm
Sep 24 07:20:56 dist-test-slave-rc1w krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:56.212177 18832 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:20:56.212618 18832 webserver.cc:460] Webserver started at http://127.3.246.126:45081/ using document root <none> and password file <none>
I20260924 07:20:56.212836 18832 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:56.212899 18832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:56.214232 18832 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:20:56.214849 18847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:56.215000 18832 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:20:56.215093 18832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "1305b1d1f3a74376abb0278f7860a18d"
format_stamp: "Formatted at 2026-09-24 07:20:54 on dist-test-slave-rc1w"
server_key: "ad40f7e39b6abc2d78f6754a43584563"
server_key_iv: "c88529992ed1c161988c328f9d2dd9e0"
server_key_version: "encryptionkey@0"
I20260924 07:20:56.215422 18832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:56.231765 18832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:56.233480 18832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:56.233635 18832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:56.237475 18832 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:43111
I20260924 07:20:56.237516 18899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:43111 every 8 connection(s)
I20260924 07:20:56.237903 18832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:56.239428 18900 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:20:56.240057 18900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Bootstrap starting.
I20260924 07:20:56.247684 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 18832
I20260924 07:20:56.248095 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:46021
--local_ip_for_outbound_sockets=127.3.246.65
--tserver_master_addrs=127.3.246.126:43111
--webserver_port=36929
--webserver_interface=127.3.246.65
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:20:56.248864 18900 log.cc:824] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:56.251497 18900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:20:56.251833 18900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d: Bootstrap complete.
I20260924 07:20:56.255038 18900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } }
I20260924 07:20:56.255293 18900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1305b1d1f3a74376abb0278f7860a18d, State: Initialized, Role: FOLLOWER
I20260924 07:20:56.255470 18900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } }
I20260924 07:20:56.255585 18900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:56.255640 18900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:56.255692 18900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:20:56.256697 18900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } }
I20260924 07:20:56.256829 18900 leader_election.cc:302] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1305b1d1f3a74376abb0278f7860a18d; no voters:
I20260924 07:20:56.257021 18900 leader_election.cc:288] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 07:20:56.257064 18904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 3 FOLLOWER]: Leader election won for term 3
I20260924 07:20:56.257174 18904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [term 3 LEADER]: Becoming Leader. State: Replica: 1305b1d1f3a74376abb0278f7860a18d, State: Running, Role: LEADER
I20260924 07:20:56.257301 18900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:56.257284 18904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } }
I20260924 07:20:56.257577 18904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1305b1d1f3a74376abb0278f7860a18d. Latest consensus state: current_term: 3 leader_uuid: "1305b1d1f3a74376abb0278f7860a18d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } } }
I20260924 07:20:56.257661 18904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:56.257581 18905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1305b1d1f3a74376abb0278f7860a18d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1305b1d1f3a74376abb0278f7860a18d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 43111 } } }
I20260924 07:20:56.257889 18905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1305b1d1f3a74376abb0278f7860a18d [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:56.258104 18909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:56.258941 18909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:56.259073 18909 catalog_manager.cc:1259] Loaded cluster ID: 22e3036aea534bcda916c64eb1afc287
I20260924 07:20:56.259122 18909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 07:20:56.259406 18909 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3b865ca9ad google::LogMessage::Fail() at ??:0
@ 0x7f3b865cd88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3b865ca4de google::LogMessage::Flush() at ??:0
@ 0x7f3b865cba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3b8e4eb0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f3b8e4f48b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3b884980d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3b8848d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3b892f16db start_thread at ??:0
@ 0x7f3b850c861f clone at ??:0
W20260924 07:20:56.357825 18903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:56.358009 18903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:56.358032 18903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:56.359179 18903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:56.359218 18903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:56.359244 18903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:20:56.360677 18903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:46021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=36929
--tserver_master_addrs=127.3.246.126:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:56.360908 18903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:56.361142 18903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:56.365999 18939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:56.366014 18936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:56.366032 18903 server_base.cc:1034] running on GCE node
W20260924 07:20:56.366029 18937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:56.366385 18903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:56.366647 18903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:56.367825 18903 hybrid_clock.cc:648] HybridClock initialized: now 1790234456367806 us; error 32 us; skew 500 ppm
Sep 24 07:20:56 dist-test-slave-rc1w krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:56.370286 18903 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:20:56.370842 18903 webserver.cc:460] Webserver started at http://127.3.246.65:36929/ using document root <none> and password file <none>
I20260924 07:20:56.371142 18903 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:56.371214 18903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:56.372740 18903 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:56.373569 18946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:56.373773 18903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:56.373864 18903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "651bd68fbf3d496c9dc50e8af8461e66"
format_stamp: "Formatted at 2026-09-24 07:20:54 on dist-test-slave-rc1w"
server_key: "bc3788d2c75c8a70e639680238efc587"
server_key_iv: "a553ea6bd34c46427b73b130c75a0b2f"
server_key_version: "encryptionkey@0"
I20260924 07:20:56.374243 18903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:56.384086 18903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:56.386104 18903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:56.386250 18903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:56.386454 18903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:56.386932 18953 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:20:56.388636 18903 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:20:56.388697 18903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 07:20:56.388741 18903 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:20:56.389341 18903 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:20:56.389391 18903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 07:20:56.389463 18953 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66: Bootstrap starting.
I20260924 07:20:56.391561 18953 log.cc:824] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:56.394825 18953 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:20:56.395202 18953 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66: Bootstrap complete.
I20260924 07:20:56.395402 18953 ts_tablet_manager.cc:1397] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:20:56.396624 18903 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:46021
I20260924 07:20:56.396659 19061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:46021 every 8 connection(s)
I20260924 07:20:56.396977 18903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:20:56.397198 18953 raft_consensus.cc:348] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:56.397723 18953 raft_consensus.cc:729] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651bd68fbf3d496c9dc50e8af8461e66, State: Initialized, Role: FOLLOWER
I20260924 07:20:56.397877 18953 consensus_queue.cc:260] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:56.398164 18953 ts_tablet_manager.cc:1428] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260924 07:20:56.404030 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 18903
I20260924 07:20:56.404347 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:35815
--local_ip_for_outbound_sockets=127.3.246.66
--tserver_master_addrs=127.3.246.126:43111
--webserver_port=35095
--webserver_interface=127.3.246.66
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:56.482959 19064 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:56.397608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:34207)
0924 07:20:56.397801 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.397845 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:56.397995 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:56.482735 (+ 84740us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":119,"threads_started":1}
W20260924 07:20:56.483196 19062 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:43111 (0 consecutive failures): Network error: Failed to ping master at 127.3.246.126:43111: Client connection negotiation failed: client connection to 127.3.246.126:43111: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:20:56.483528 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.483410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:56123)
0924 07:20:56.483464 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.483508 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 07:20:56.483820 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.483725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:51729)
0924 07:20:56.483768 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.483797 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 07:20:56.483927 19062 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:20:56.487759 19066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:56.487964 19066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:56.488003 19066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:56.489207 19066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:56.489271 19066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:56.489320 19066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:20:56.490823 19066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:35815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=35095
--tserver_master_addrs=127.3.246.126:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:56.491077 19066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:56.491324 19066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:56.496356 19074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:56.496367 19072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:56.496632 19071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:56.496680 19066 server_base.cc:1034] running on GCE node
I20260924 07:20:56.496935 19066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:56.497210 19066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:56.498343 19066 hybrid_clock.cc:648] HybridClock initialized: now 1790234456498327 us; error 44 us; skew 500 ppm
Sep 24 07:20:56 dist-test-slave-rc1w krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:56.501441 19066 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:20:56.502174 19066 webserver.cc:460] Webserver started at http://127.3.246.66:35095/ using document root <none> and password file <none>
I20260924 07:20:56.502444 19066 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:56.502517 19066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:56.504343 19066 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:20:56.505249 19081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:56.505401 19066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:20:56.505496 19066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "e24f48a3e33449ed9d349aa731765fc4"
format_stamp: "Formatted at 2026-09-24 07:20:55 on dist-test-slave-rc1w"
server_key: "03d2f076bee418ce3e2bc20f2abc0481"
server_key_iv: "54868eb8b1d9ae3fe489a876a8a9be81"
server_key_version: "encryptionkey@0"
I20260924 07:20:56.505897 19066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:56.513244 19066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:56.515431 19066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:56.515594 19066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:56.515812 19066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:56.516240 19088 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:20:56.517640 19066 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:20:56.517729 19066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 07:20:56.517773 19066 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:20:56.518363 19066 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:20:56.518414 19066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 07:20:56.518473 19088 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: Bootstrap starting.
I20260924 07:20:56.525404 19066 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:35815
I20260924 07:20:56.525508 19195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:35815 every 8 connection(s)
I20260924 07:20:56.525856 19066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:20:56.526945 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.526484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:44239)
0924 07:20:56.526677 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.526771 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
W20260924 07:20:56.527182 19196 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:43111 (0 consecutive failures): Network error: Failed to ping master at 127.3.246.126:43111: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
I20260924 07:20:56.527459 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.527352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:42219)
0924 07:20:56.527409 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.527435 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 07:20:56.527766 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.527703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:42221)
0924 07:20:56.527734 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.527753 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 07:20:56.527880 19196 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:20:56.528756 19088 log.cc:824] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:56.529457 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 19066
I20260924 07:20:56.529758 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:34499
--local_ip_for_outbound_sockets=127.3.246.67
--tserver_master_addrs=127.3.246.126:43111
--webserver_port=39971
--webserver_interface=127.3.246.67
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:20:56.530895 19088 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:20:56.531177 19088 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: Bootstrap complete.
I20260924 07:20:56.532248 19088 ts_tablet_manager.cc:1397] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20260924 07:20:56.534514 19088 raft_consensus.cc:348] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:56.534963 19088 raft_consensus.cc:729] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e24f48a3e33449ed9d349aa731765fc4, State: Initialized, Role: FOLLOWER
I20260924 07:20:56.535123 19088 consensus_queue.cc:260] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:56.535421 19088 ts_tablet_manager.cc:1428] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20260924 07:20:56.535698 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.535572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:43079)
0924 07:20:56.535631 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.535670 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 07:20:56.536021 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.535912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:40747)
0924 07:20:56.535964 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.535995 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 07:20:56.611861 19200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:56.612078 19200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:56.612103 19200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:56.613245 19200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:56.613287 19200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:56.613341 19200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:20:56.614775 19200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:34499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=39971
--tserver_master_addrs=127.3.246.126:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:20:56.615005 19200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:56.615233 19200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:20:56.620208 19200 server_base.cc:1034] running on GCE node
W20260924 07:20:56.620106 19206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:56.620107 19207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:56.620126 19209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:56.620563 19200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:56.620816 19200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:56.621989 19200 hybrid_clock.cc:648] HybridClock initialized: now 1790234456621976 us; error 37 us; skew 500 ppm
Sep 24 07:20:56 dist-test-slave-rc1w krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:56.624306 19200 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:20:56.624747 19200 webserver.cc:460] Webserver started at http://127.3.246.67:39971/ using document root <none> and password file <none>
I20260924 07:20:56.624969 19200 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:56.625034 19200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:56.626365 19200 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:20:56.626924 19216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:56.627082 19200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:20:56.627178 19200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "da63ac1c49c741448c869fd8bbe50544"
format_stamp: "Formatted at 2026-09-24 07:20:55 on dist-test-slave-rc1w"
server_key: "77b8c8b23a156c5a130da222e59ff850"
server_key_iv: "22995d7f0351a35b2732f5c6d351cb91"
server_key_version: "encryptionkey@0"
I20260924 07:20:56.627506 19200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:56.637565 19200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:56.639412 19200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:56.639573 19200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:56.639802 19200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:56.640120 19200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:56.640255 19200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:56.640352 19200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:56.640383 19200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:56.646524 19200 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:34499
I20260924 07:20:56.646574 19329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:34499 every 8 connection(s)
I20260924 07:20:56.646906 19200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:20:56.647970 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.647551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:58011)
0924 07:20:56.647751 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.647826 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
W20260924 07:20:56.648411 19330 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:43111 (0 consecutive failures): Network error: Failed to ping master at 127.3.246.126:43111: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
I20260924 07:20:56.648670 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.648586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:35805)
0924 07:20:56.648626 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.648652 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 07:20:56.648942 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:56.648873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:39567)
0924 07:20:56.648909 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:56.648929 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 07:20:56.649053 19330 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:20:56.654865 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 19200
I20260924 07:20:57.484501 19334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:57.484278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:39497)
0924 07:20:57.484400 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:57.484437 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260924 07:20:57.536614 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:57.536385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:57245)
0924 07:20:57.536533 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:57.536568 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
I20260924 07:20:57.649580 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:57.649380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:48847)
0924 07:20:57.649493 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:57.649530 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260924 07:20:57.673280 19337 raft_consensus.cc:482] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:57.673399 19337 raft_consensus.cc:504] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:57.673831 19337 leader_election.cc:288] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da63ac1c49c741448c869fd8bbe50544 (127.3.246.67:34499), 651bd68fbf3d496c9dc50e8af8461e66 (127.3.246.65:46021)
Sep 24 07:20:57 dist-test-slave-rc1w krb5kdc[17763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.67@KRBTEST.COM
Sep 24 07:20:57 dist-test-slave-rc1w krb5kdc[17763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.65@KRBTEST.COM
I20260924 07:20:57.684168 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:57.674070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:52305 (local address 127.3.246.67:34499)
0924 07:20:57.674222 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:57.674234 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:57.674267 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:20:57.674778 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:57.674780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:57.674817 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:57.674956 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:57.675404 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:57.676268 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:57.677318 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:57.677832 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:57.677957 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:57.680355 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:57.680358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:57.680358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:57.680368 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:57.682340 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:57.682778 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:57.682780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:57.682780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:57.682886 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:57.683172 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:57.683173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:57.683173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:57.683495 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:57.683581 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:57.683901 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 07:20:57.684051 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":42,"threads_started":1}
I20260924 07:20:57.684161 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:57.674021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:34499 (local address 127.3.246.66:52305)
0924 07:20:57.674129 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:57.674139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:57.674281 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:57.674950 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:57.674951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:57.675201 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:57.675287 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:57.675293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:57.676406 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:57.676407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:57.677217 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:57.677219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:57.677931 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:57.677932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:57.678053 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:57.678671 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:57.678681 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:57.680246 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:57.682416 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:57.682422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:57.682424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:57.682697 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:57.682950 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:57.682951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:57.682951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:57.683095 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:57.683566 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:57.683572 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:57.683732 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 07:20:57.683847 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 07:20:57.684040 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":54,"thread_start_us":55,"threads_started":1}
I20260924 07:20:57.684310 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:57.674013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:46021 (local address 127.3.246.66:40989)
0924 07:20:57.674072 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:57.674098 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:57.674281 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:57.674684 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:57.674688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:57.675109 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:57.675287 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:57.675293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:57.676437 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:57.676438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:57.677241 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:57.677243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:57.677953 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:57.677954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:57.678052 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:57.678855 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:57.678860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:57.680626 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:57.683045 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:57.683046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:57.683046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:57.683254 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:57.683475 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:57.683476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:57.683476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:57.683544 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:57.683960 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:57.683961 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:57.684004 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 07:20:57.684082 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:20:57.684169 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":35}
I20260924 07:20:57.684410 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:57.674174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:40989 (local address 127.3.246.65:46021)
0924 07:20:57.674446 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:57.674450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:57.674459 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:20:57.674529 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:57.674530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:57.674548 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:57.674649 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:57.675403 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:57.676282 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:57.677329 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:57.677860 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:57.678015 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:57.680748 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:57.680751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:57.680751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:57.680760 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:57.682972 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:57.683335 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:57.683338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:57.683338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:57.683416 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:57.683607 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:57.683608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:57.683608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:57.683893 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:57.683967 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:57.684151 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 07:20:57.684282 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":65,"threads_started":1}
I20260924 07:20:57.684825 19016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e97d352bdd405282d360ecfa962a8f" candidate_uuid: "e24f48a3e33449ed9d349aa731765fc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "651bd68fbf3d496c9dc50e8af8461e66" is_pre_election: true
I20260924 07:20:57.684980 19016 raft_consensus.cc:2454] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e24f48a3e33449ed9d349aa731765fc4 in term 1.
I20260924 07:20:57.685246 19085 leader_election.cc:302] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 651bd68fbf3d496c9dc50e8af8461e66, e24f48a3e33449ed9d349aa731765fc4; no voters:
I20260924 07:20:57.685381 19337 raft_consensus.cc:2781] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 07:20:57.685456 19337 raft_consensus.cc:482] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:20:57.685494 19337 raft_consensus.cc:3037] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 1 FOLLOWER]: Advancing to term 2
W20260924 07:20:57.685755 19084 leader_election.cc:341] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da63ac1c49c741448c869fd8bbe50544 (127.3.246.67:34499): Not found: Tablet not found: 11e97d352bdd405282d360ecfa962a8f
I20260924 07:20:57.686725 19337 raft_consensus.cc:504] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:57.686872 19337 leader_election.cc:288] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [CANDIDATE]: Term 2 election: Requested vote from peers da63ac1c49c741448c869fd8bbe50544 (127.3.246.67:34499), 651bd68fbf3d496c9dc50e8af8461e66 (127.3.246.65:46021)
W20260924 07:20:57.687106 19084 leader_election.cc:341] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer da63ac1c49c741448c869fd8bbe50544 (127.3.246.67:34499): Not found: Tablet not found: 11e97d352bdd405282d360ecfa962a8f
I20260924 07:20:57.687081 19016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e97d352bdd405282d360ecfa962a8f" candidate_uuid: "e24f48a3e33449ed9d349aa731765fc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "651bd68fbf3d496c9dc50e8af8461e66"
I20260924 07:20:57.687163 19016 raft_consensus.cc:3037] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:20:57.688412 19016 raft_consensus.cc:2454] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e24f48a3e33449ed9d349aa731765fc4 in term 2.
I20260924 07:20:57.688612 19085 leader_election.cc:302] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 651bd68fbf3d496c9dc50e8af8461e66, e24f48a3e33449ed9d349aa731765fc4; no voters: da63ac1c49c741448c869fd8bbe50544
I20260924 07:20:57.688737 19337 raft_consensus.cc:2781] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:20:57.688856 19337 raft_consensus.cc:686] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [term 2 LEADER]: Becoming Leader. State: Replica: e24f48a3e33449ed9d349aa731765fc4, State: Running, Role: LEADER
I20260924 07:20:57.688952 19337 consensus_queue.cc:237] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:57.689234 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:57.689101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:52861)
0924 07:20:57.689173 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:57.689204 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 07:20:58.110201 19016 raft_consensus.cc:1261] T 11e97d352bdd405282d360ecfa962a8f P 651bd68fbf3d496c9dc50e8af8461e66 [term 2 FOLLOWER]: Refusing update from remote peer e24f48a3e33449ed9d349aa731765fc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260924 07:20:58.110574 19337 consensus_queue.cc:1035] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 07:20:58.111515 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:58.111347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:40511)
0924 07:20:58.111444 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:58.111472 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":41,"threads_started":1}
I20260924 07:20:58.112253 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:58.112151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:37467)
0924 07:20:58.112191 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:58.112218 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 07:20:58.112463 19084 consensus_peers.cc:487] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4 -> Peer da63ac1c49c741448c869fd8bbe50544 (127.3.246.67:34499): Couldn't send request to peer da63ac1c49c741448c869fd8bbe50544. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11e97d352bdd405282d360ecfa962a8f. This is attempt 1: this message will repeat every 5th retry.
I20260924 07:20:58.112617 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:58.112488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:60749)
0924 07:20:58.112548 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:58.112590 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:20:58.593448 19349 ts_tablet_manager.cc:927] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Initiating tablet copy from peer e24f48a3e33449ed9d349aa731765fc4 (127.3.246.66:35815)
I20260924 07:20:58.593693 19349 tablet_copy_client.cc:316] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: tablet copy: Beginning tablet copy session from remote peer at address 127.3.246.66:35815
Sep 24 07:20:58 dist-test-slave-rc1w krb5kdc[17763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.66@KRBTEST.COM
I20260924 07:20:58.601462 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:58.594106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:58631 (local address 127.3.246.66:35815)
0924 07:20:58.594245 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:58.594250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:58.594262 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:20:58.594327 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:58.594329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:58.594344 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:58.594418 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:58.594902 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:58.595594 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:58.596385 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:58.596685 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:58.596773 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:58.598813 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:58.598820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:58.598821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:58.598828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:58.600208 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:58.600606 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:58.600609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:58.600609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:58.600668 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:58.600862 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:58.600863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:58.600863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:58.601015 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:58.601106 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:58.601274 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 07:20:58.601360 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260924 07:20:58.601480 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:58.593996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:35815 (local address 127.3.246.67:58631)
0924 07:20:58.594153 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:58.594169 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:58.594259 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:58.594439 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:58.594445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:58.594755 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:58.594816 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:58.594818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:58.595681 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:58.595683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:58.596285 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:58.596287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:58.596774 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:58.596775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:58.596865 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:58.597350 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:58.597354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:58.598727 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:58.600304 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:58.600310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:58.600313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:58.600514 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:58.600736 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:58.600737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:58.600737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:58.600799 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:58.601085 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:58.601092 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:58.601144 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 07:20:58.601241 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 07:20:58.601363 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260924 07:20:58.601948 19170 tablet_copy_service.cc:140] P e24f48a3e33449ed9d349aa731765fc4: Received BeginTabletCopySession request for tablet 11e97d352bdd405282d360ecfa962a8f from peer da63ac1c49c741448c869fd8bbe50544 ({username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:58631)
I20260924 07:20:58.602010 19170 tablet_copy_service.cc:161] P e24f48a3e33449ed9d349aa731765fc4: Beginning new tablet copy session on tablet 11e97d352bdd405282d360ecfa962a8f from peer da63ac1c49c741448c869fd8bbe50544 at {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:58631: session id = da63ac1c49c741448c869fd8bbe50544-11e97d352bdd405282d360ecfa962a8f
I20260924 07:20:58.602485 19170 tablet_copy_source_session.cc:205] T 11e97d352bdd405282d360ecfa962a8f P e24f48a3e33449ed9d349aa731765fc4: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 07:20:58.603425 19349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e97d352bdd405282d360ecfa962a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:58.606098 19349 tablet_copy_client.cc:791] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: tablet copy: Starting download of 0 data blocks...
I20260924 07:20:58.606271 19349 tablet_copy_client.cc:655] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: tablet copy: Starting download of 1 WAL segments...
I20260924 07:20:58.607079 19349 tablet_copy_client.cc:531] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 07:20:58.607805 19349 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Bootstrap starting.
I20260924 07:20:58.609438 19349 log.cc:824] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:58.611835 19349 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:20:58.612119 19349 tablet_bootstrap.cc:492] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Bootstrap complete.
I20260924 07:20:58.612247 19349 ts_tablet_manager.cc:1397] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260924 07:20:58.613240 19349 raft_consensus.cc:348] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:58.613376 19349 raft_consensus.cc:729] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: da63ac1c49c741448c869fd8bbe50544, State: Initialized, Role: FOLLOWER
I20260924 07:20:58.613503 19349 consensus_queue.cc:260] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24f48a3e33449ed9d349aa731765fc4" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35815 } } peers { permanent_uuid: "da63ac1c49c741448c869fd8bbe50544" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34499 } } peers { permanent_uuid: "651bd68fbf3d496c9dc50e8af8461e66" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46021 } }
I20260924 07:20:58.613780 19349 ts_tablet_manager.cc:1428] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20260924 07:20:58.614123 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:58.614003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:54981)
0924 07:20:58.614056 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:58.614092 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 07:20:58.614197 19170 tablet_copy_service.cc:342] P e24f48a3e33449ed9d349aa731765fc4: Request end of tablet copy session da63ac1c49c741448c869fd8bbe50544-11e97d352bdd405282d360ecfa962a8f received from {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:58631
I20260924 07:20:58.614259 19170 tablet_copy_service.cc:433] P e24f48a3e33449ed9d349aa731765fc4: ending tablet copy session da63ac1c49c741448c869fd8bbe50544-11e97d352bdd405282d360ecfa962a8f on tablet 11e97d352bdd405282d360ecfa962a8f with peer da63ac1c49c741448c869fd8bbe50544
I20260924 07:20:58.614454 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:58.614343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:56917)
0924 07:20:58.614400 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:58.614429 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 07:20:59.056294 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:59.056069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:45089)
0924 07:20:59.056196 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:59.056245 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260924 07:20:59.056223 19284 raft_consensus.cc:1203] T 11e97d352bdd405282d360ecfa962a8f P da63ac1c49c741448c869fd8bbe50544 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260924 07:20:59.056790 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:59.056678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:59353)
0924 07:20:59.056735 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:59.056761 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 07:20:59.113184 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:59.112990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:58683)
0924 07:20:59.113105 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:59.113139 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":51,"threads_started":1}
I20260924 07:21:00.057013 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:00.056777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:57079)
0924 07:21:00.056925 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:00.056967 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260924 07:21:00.057495 19360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:00.057297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:38747)
0924 07:21:00.057397 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:00.057432 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":49,"threads_started":1}
I20260924 07:21:00.114089 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:00.113855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:57099)
0924 07:21:00.114006 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:00.114040 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260924 07:21:01.057927 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:01.057569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:56519)
0924 07:21:01.057737 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:01.057795 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260924 07:21:01.058241 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:01.057972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:39659)
0924 07:21:01.058142 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:01.058188 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
I20260924 07:21:01.114799 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:01.114569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:59103)
0924 07:21:01.114704 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:01.114744 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260924 07:21:02.058729 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:02.058527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:37171)
0924 07:21:02.058651 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:02.058683 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260924 07:21:02.058943 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:02.058716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:39149)
0924 07:21:02.058836 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:02.058894 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260924 07:21:02.059157 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:02.059073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:33069)
0924 07:21:02.059107 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:02.059137 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 07:21:02.115546 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:02.115328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:36551)
0924 07:21:02.115446 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:02.115491 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260924 07:21:03.059705 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:03.059463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:51087)
0924 07:21:03.059611 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:03.059656 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260924 07:21:03.059733 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:03.059549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:54057)
0924 07:21:03.059659 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:03.059692 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":47,"threads_started":1}
I20260924 07:21:03.116309 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:03.116017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:44303)
0924 07:21:03.116167 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:03.116249 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260924 07:21:04.060492 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:04.060271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:41987)
0924 07:21:04.060411 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:04.060445 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260924 07:21:04.060501 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:04.060279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:52099)
0924 07:21:04.060419 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:04.060461 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260924 07:21:04.117129 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:04.116886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:44871)
0924 07:21:04.117034 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:04.117082 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260924 07:21:05.061168 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:05.060952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:32825)
0924 07:21:05.061088 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:05.061121 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260924 07:21:05.061194 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:05.060989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:59053)
0924 07:21:05.061123 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:05.061153 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260924 07:21:05.117859 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:05.117619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:54389)
0924 07:21:05.117769 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:05.117809 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260924 07:21:06.061916 19377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:06.061669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:44865)
0924 07:21:06.061811 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:06.061849 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260924 07:21:06.061915 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:06.061707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:38009)
0924 07:21:06.061833 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:06.061873 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":51,"threads_started":1}
I20260924 07:21:06.118510 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:06.118299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:47637)
0924 07:21:06.118430 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:06.118464 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260924 07:21:07.062634 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:07.062428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:60257)
0924 07:21:07.062558 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:07.062588 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260924 07:21:07.062685 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:07.062482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:47133)
0924 07:21:07.062605 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:07.062644 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260924 07:21:07.119215 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:07.118984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:38433)
0924 07:21:07.119128 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:07.119168 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260924 07:21:08.063385 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:08.063159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:37939)
0924 07:21:08.063315 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:08.063344 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260924 07:21:08.063383 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:08.063159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:51959)
0924 07:21:08.063293 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:08.063335 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260924 07:21:08.119848 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:08.119628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:40747)
0924 07:21:08.119761 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:08.119799 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260924 07:21:09.064138 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:09.063907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:36179)
0924 07:21:09.064070 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:09.064095 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260924 07:21:09.064138 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:09.063907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:36821)
0924 07:21:09.064044 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:09.064082 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20260924 07:21:09.120564 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:09.120341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:44551)
0924 07:21:09.120475 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:09.120514 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":58,"threads_started":1}
I20260924 07:21:10.064864 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:10.064628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:45423)
0924 07:21:10.064753 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:10.064792 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260924 07:21:10.064877 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:10.064674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:51061)
0924 07:21:10.064798 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:10.064836 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260924 07:21:10.121392 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:10.121151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:45303)
0924 07:21:10.121298 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:10.121345 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
I20260924 07:21:11.065640 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:11.065427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.66:53651)
0924 07:21:11.065560 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:11.065594 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260924 07:21:11.065653 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:11.065464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.67:59759)
0924 07:21:11.065578 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:11.065614 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260924 07:21:11.122151 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:11.121901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:43111 (local address 127.3.246.65:45135)
0924 07:21:11.122047 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:11.122091 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260924 07:21:11.659215 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 18903
I20260924 07:21:11.663760 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 19066
I20260924 07:21:11.668583 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 19200
2026-09-24T07:21:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18144 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:21:11 dist-test-slave-rc1w krb5kdc[19398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:21:11 dist-test-slave-rc1w krb5kdc[19398](info): set up 2 sockets
Sep 24 07:21:11 dist-test-slave-rc1w krb5kdc[19398](info): commencing operation
krb5kdc: starting...
W20260924 07:21:12.698001 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:21:12 dist-test-slave-rc1w krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:21:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:21:12.817291 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:42205
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:42205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:21:12.896564 19411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:21:12.896814 19411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:21:12.896849 19411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:21:12.898078 19411 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:21:12.898138 19411 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:21:12.898159 19411 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:21:12.898176 19411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:21:12.898203 19411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:21:12.898236 19411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:21:12.898262 19411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:21:12.898288 19411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:21:12.899787 19411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:42205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:42205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:21:12.900066 19411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:21:12.900295 19411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:21:12.905539 19416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:12.905546 19417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:12.905570 19411 server_base.cc:1034] running on GCE node
W20260924 07:21:12.905539 19419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:12.906035 19411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:21:12.906338 19411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:21:12.907505 19411 hybrid_clock.cc:648] HybridClock initialized: now 1790234472907481 us; error 42 us; skew 500 ppm
Sep 24 07:21:12 dist-test-slave-rc1w krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:21:12.910125 19411 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:21:12.910570 19411 webserver.cc:460] Webserver started at http://127.3.246.126:34191/ using document root <none> and password file <none>
I20260924 07:21:12.910801 19411 fs_manager.cc:359] Metadata directory not provided
I20260924 07:21:12.910847 19411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:21:12.910995 19411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:21:12.912130 19411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "e430d6416d454b74b9981d6def6d33a8"
format_stamp: "Formatted at 2026-09-24 07:21:12 on dist-test-slave-rc1w"
server_key: "dff6069f8e31002db52513ae325abb50"
server_key_iv: "15098e55d39bf04561ceb85625d0b8a5"
server_key_version: "encryptionkey@0"
I20260924 07:21:12.912500 19411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "e430d6416d454b74b9981d6def6d33a8"
format_stamp: "Formatted at 2026-09-24 07:21:12 on dist-test-slave-rc1w"
server_key: "dff6069f8e31002db52513ae325abb50"
server_key_iv: "15098e55d39bf04561ceb85625d0b8a5"
server_key_version: "encryptionkey@0"
I20260924 07:21:12.913928 19411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:21:12.914638 19426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:12.914805 19411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:21:12.914914 19411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "e430d6416d454b74b9981d6def6d33a8"
format_stamp: "Formatted at 2026-09-24 07:21:12 on dist-test-slave-rc1w"
server_key: "dff6069f8e31002db52513ae325abb50"
server_key_iv: "15098e55d39bf04561ceb85625d0b8a5"
server_key_version: "encryptionkey@0"
I20260924 07:21:12.914989 19411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:21:12.950398 19411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:21:12.952432 19411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:21:12.952605 19411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:21:12.956707 19411 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:42205
I20260924 07:21:12.956745 19478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:42205 every 8 connection(s)
I20260924 07:21:12.957144 19411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:21:12.958137 19479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:21:12.960412 19479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Bootstrap starting.
I20260924 07:21:12.960968 19479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Neither blocks nor log segments found. Creating new log.
I20260924 07:21:12.961208 19479 log.cc:824] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Log is configured to *not* fsync() on all Append() calls
I20260924 07:21:12.961930 19479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: No bootstrap required, opened a new log
I20260924 07:21:12.962181 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 19411
I20260924 07:21:12.962288 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:21:12.962492 4057 external_mini_cluster.cc:1428] Setting key f5dc2cb5a41b2a079f0f39841870917a
I20260924 07:21:12.964047 19479 raft_consensus.cc:348] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } }
I20260924 07:21:12.964208 19479 raft_consensus.cc:374] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:21:12.964246 19479 raft_consensus.cc:729] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e430d6416d454b74b9981d6def6d33a8, State: Initialized, Role: FOLLOWER
I20260924 07:21:12.964417 19479 consensus_queue.cc:260] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } }
I20260924 07:21:12.964512 19479 raft_consensus.cc:388] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:21:12.964550 19479 raft_consensus.cc:482] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:21:12.964600 19479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:21:12.965548 19479 raft_consensus.cc:504] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } }
I20260924 07:21:12.965637 19479 leader_election.cc:302] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e430d6416d454b74b9981d6def6d33a8; no voters:
I20260924 07:21:12.965860 19479 leader_election.cc:288] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:21:12.965879 19484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:21:12.965988 19484 raft_consensus.cc:686] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 1 LEADER]: Becoming Leader. State: Replica: e430d6416d454b74b9981d6def6d33a8, State: Running, Role: LEADER
I20260924 07:21:12.966068 19484 consensus_queue.cc:237] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } }
I20260924 07:21:12.966128 19479 sys_catalog.cc:564] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:21:12.966389 19484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e430d6416d454b74b9981d6def6d33a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } } }
I20260924 07:21:12.966573 19484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: This master's current role is: LEADER
I20260924 07:21:12.966766 19484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e430d6416d454b74b9981d6def6d33a8. Latest consensus state: current_term: 1 leader_uuid: "e430d6416d454b74b9981d6def6d33a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } } }
I20260924 07:21:12.966835 19484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: This master's current role is: LEADER
I20260924 07:21:12.967186 19491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:21:12.967759 19491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:21:12.969420 19491 catalog_manager.cc:1347] Generated new cluster ID: 41efdacd3a2a417684807662d661e034
I20260924 07:21:12.969475 19491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:21:12 dist-test-slave-rc1w krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:21:12.974848 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:12.963015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.3.246.126:42205)
0924 07:21:12.963196 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:12.963205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:12.963240 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:21:12.964028 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:12.964045 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:12.964094 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:12.964231 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:21:12.965045 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:12.966216 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:12.967361 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:12.967978 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:12.968088 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:12.971022 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:21:12.971026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:21:12.971026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:21:12.971037 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:21:12.973074 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:12.973553 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:12.973555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:12.973556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:12.973667 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:12.973900 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:12.973901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:12.973901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:12.974203 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:21:12.974338 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:12.974519 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 07:21:12.974648 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260924 07:21:12.979115 19491 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:21:12.979642 19491 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:21:12.988287 19491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Generated new TSK 0
I20260924 07:21:12.988533 19491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:21:13.017118 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42205
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:21:13.099455 19505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:21:13.099661 19505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:21:13.099684 19505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:21:13.100854 19505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:21:13.100900 19505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:21:13.100927 19505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:21:13.102807 19505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:21:13.103091 19505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:21:13.103371 19505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:21:13.108594 19513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:13.108601 19511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:13.108594 19510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:13.108968 19505 server_base.cc:1034] running on GCE node
I20260924 07:21:13.109205 19505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:21:13.109405 19505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:21:13.110554 19505 hybrid_clock.cc:648] HybridClock initialized: now 1790234473110540 us; error 27 us; skew 500 ppm
Sep 24 07:21:13 dist-test-slave-rc1w krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:21:13.113183 19505 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:21:13.113641 19505 webserver.cc:460] Webserver started at http://127.3.246.65:39737/ using document root <none> and password file <none>
I20260924 07:21:13.113945 19505 fs_manager.cc:359] Metadata directory not provided
I20260924 07:21:13.114001 19505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:21:13.114140 19505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:21:13.115216 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "94efd2fe59c246109c6b64ff2411d533"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "2fba192a80780e2cd4083be9840dca9d"
server_key_iv: "20d4b1757b16e9e6d3cc01cec1fb0c47"
server_key_version: "encryptionkey@0"
I20260924 07:21:13.115574 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "94efd2fe59c246109c6b64ff2411d533"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "2fba192a80780e2cd4083be9840dca9d"
server_key_iv: "20d4b1757b16e9e6d3cc01cec1fb0c47"
server_key_version: "encryptionkey@0"
I20260924 07:21:13.116950 19505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:21:13.117811 19520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:13.118010 19505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:21:13.118091 19505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "94efd2fe59c246109c6b64ff2411d533"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "2fba192a80780e2cd4083be9840dca9d"
server_key_iv: "20d4b1757b16e9e6d3cc01cec1fb0c47"
server_key_version: "encryptionkey@0"
I20260924 07:21:13.118144 19505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:21:13.161302 19505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:21:13.163825 19505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:21:13.163949 19505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:21:13.164148 19505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:21:13.164479 19505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:21:13.164516 19505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:13.164570 19505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:21:13.164595 19505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:13.170806 19505 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:42187
I20260924 07:21:13.170892 19633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:42187 every 8 connection(s)
I20260924 07:21:13.171217 19505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:21:13.172420 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 19505
I20260924 07:21:13.172541 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:21:13.172742 4057 external_mini_cluster.cc:1428] Setting key 05903300aa522406fe2211c3ae27e0b7
Sep 24 07:21:13 dist-test-slave-rc1w krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
I20260924 07:21:13.181348 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.171965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:34827 (local address 127.3.246.126:42205)
0924 07:21:13.172031 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:13.172033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:13.172278 (+ 245us) server_negotiation.cc:407] Connection header received
0924 07:21:13.172931 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:13.172933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:13.172950 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:13.173002 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:21:13.174104 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.174609 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.175648 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.175885 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.175972 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.178325 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:21:13.178327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:21:13.178327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:21:13.178338 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:21:13.179879 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:13.180336 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:13.180337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:13.180337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:13.180397 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:13.180666 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:13.180666 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:13.180667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:13.180803 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:21:13.180890 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:13.181161 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 07:21:13.181245 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:21:13.181602 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.171932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:34827)
0924 07:21:13.172142 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:13.172177 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:13.172820 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:13.173088 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:13.173092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:13.173809 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:21:13.174006 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.174011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.174707 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.174709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.175556 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.175559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.175983 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.175984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.176116 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.176636 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:21:13.176644 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:21:13.178225 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:21:13.180021 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:13.180029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:13.180031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:13.180256 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:13.180479 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:13.180480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:13.180480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:13.180591 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:13.180874 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:21:13.180880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:21:13.181025 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 07:21:13.181139 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 07:21:13.181297 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":122,"threads_started":1}
I20260924 07:21:13.182147 19634 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42205
I20260924 07:21:13.182250 19634 heartbeater.cc:464] Registering TS with master...
I20260924 07:21:13.182466 19634 heartbeater.cc:511] Master 127.3.246.126:42205 requested a full tablet report, sending...
I20260924 07:21:13.183115 19443 ts_manager.cc:194] Registered new tserver with Master: 94efd2fe59c246109c6b64ff2411d533 (127.3.246.65:42187)
I20260924 07:21:13.183781 19443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:34827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:21:13.201346 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42205
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:21:13.285785 19639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:21:13.286026 19639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:21:13.286063 19639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:21:13.287186 19639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:21:13.287256 19639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:21:13.287302 19639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:21:13.288767 19639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:21:13.289034 19639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:21:13.289258 19639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:21:13.294232 19644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:13.294283 19647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:13.294450 19639 server_base.cc:1034] running on GCE node
W20260924 07:21:13.294485 19645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:13.294723 19639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:21:13.294988 19639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:21:13.296155 19639 hybrid_clock.cc:648] HybridClock initialized: now 1790234473296141 us; error 27 us; skew 500 ppm
Sep 24 07:21:13 dist-test-slave-rc1w krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:21:13.298695 19639 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:21:13.299168 19639 webserver.cc:460] Webserver started at http://127.3.246.66:39453/ using document root <none> and password file <none>
I20260924 07:21:13.299397 19639 fs_manager.cc:359] Metadata directory not provided
I20260924 07:21:13.299474 19639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:21:13.299631 19639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:21:13.300671 19639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "a0e9282915e448b1aa06900beb8f61d6"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "a3cb4c01aa85201b916706371249c356"
server_key_iv: "8e68d6373ff72a6d064d1ffe7360d772"
server_key_version: "encryptionkey@0"
I20260924 07:21:13.301041 19639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "a0e9282915e448b1aa06900beb8f61d6"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "a3cb4c01aa85201b916706371249c356"
server_key_iv: "8e68d6373ff72a6d064d1ffe7360d772"
server_key_version: "encryptionkey@0"
I20260924 07:21:13.302512 19639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:21:13.303272 19654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:13.303447 19639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:13.303555 19639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "a0e9282915e448b1aa06900beb8f61d6"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "a3cb4c01aa85201b916706371249c356"
server_key_iv: "8e68d6373ff72a6d064d1ffe7360d772"
server_key_version: "encryptionkey@0"
I20260924 07:21:13.303640 19639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:21:13.317524 19639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:21:13.319681 19639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:21:13.319841 19639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:21:13.320072 19639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:21:13.320384 19639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:21:13.320436 19639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:13.320484 19639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:21:13.320513 19639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:13.326251 19639 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:36999
I20260924 07:21:13.326292 19767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:36999 every 8 connection(s)
I20260924 07:21:13.326606 19639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:21:13 dist-test-slave-rc1w krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:21:13.335811 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.327334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:52895 (local address 127.3.246.126:42205)
0924 07:21:13.327379 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:13.327381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:13.327572 (+ 191us) server_negotiation.cc:407] Connection header received
0924 07:21:13.328110 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:13.328111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:13.328126 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:13.328177 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:21:13.329106 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.329621 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.330656 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.330873 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.330960 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.333275 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:21:13.333276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:21:13.333277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:21:13.333283 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:21:13.334571 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:13.334943 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:13.334944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:13.334944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:13.335009 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:13.335271 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:13.335271 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:13.335272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:13.335370 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:21:13.335419 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:13.335682 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 07:21:13.335748 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:21:13.336094 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.327294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:52895)
0924 07:21:13.327450 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:13.327489 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:13.327996 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:13.328243 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:13.328247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:13.328810 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:21:13.328995 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.328999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.329756 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.329757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.330559 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.330561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.330970 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.330971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.331097 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.331659 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:21:13.331668 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:21:13.333190 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:21:13.334642 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:13.334649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:13.334650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:13.334875 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:13.335090 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:13.335091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:13.335091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:13.335205 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:13.335429 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:21:13.335434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:21:13.335567 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 07:21:13.335660 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 07:21:13.335790 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260924 07:21:13.336418 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 19639
I20260924 07:21:13.336550 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:21:13.336673 19768 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42205
I20260924 07:21:13.336772 4057 external_mini_cluster.cc:1428] Setting key 89e1662b80af0a31bb4d2c1d3863e97c
I20260924 07:21:13.336767 19768 heartbeater.cc:464] Registering TS with master...
I20260924 07:21:13.336952 19768 heartbeater.cc:511] Master 127.3.246.126:42205 requested a full tablet report, sending...
I20260924 07:21:13.337513 19443 ts_manager.cc:194] Registered new tserver with Master: a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999)
I20260924 07:21:13.337956 19443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:52895
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:21:13.364320 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42205
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:21:13.445477 19773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:21:13.445696 19773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:21:13.445720 19773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:21:13.446822 19773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:21:13.446858 19773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:21:13.446883 19773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:21:13.448254 19773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:42205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:21:13.448621 19773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:21:13.448841 19773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:21:13.453871 19779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:13.453877 19778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:13.453877 19781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:13.454121 19773 server_base.cc:1034] running on GCE node
I20260924 07:21:13.454370 19773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:21:13.454612 19773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:21:13.455780 19773 hybrid_clock.cc:648] HybridClock initialized: now 1790234473455764 us; error 31 us; skew 500 ppm
Sep 24 07:21:13 dist-test-slave-rc1w krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:21:13.458331 19773 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:21:13.458765 19773 webserver.cc:460] Webserver started at http://127.3.246.67:38177/ using document root <none> and password file <none>
I20260924 07:21:13.458992 19773 fs_manager.cc:359] Metadata directory not provided
I20260924 07:21:13.459059 19773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:21:13.459208 19773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:21:13.460237 19773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "887681cd91e948d5aa7024c0f3a644c4"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "c839efa65e9577576f69116c616a719d"
server_key_iv: "31268a15b90b9fd3d7023dd80589e299"
server_key_version: "encryptionkey@0"
I20260924 07:21:13.460592 19773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "887681cd91e948d5aa7024c0f3a644c4"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "c839efa65e9577576f69116c616a719d"
server_key_iv: "31268a15b90b9fd3d7023dd80589e299"
server_key_version: "encryptionkey@0"
I20260924 07:21:13.461961 19773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:21:13.462658 19788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:13.462831 19773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:21:13.462941 19773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "887681cd91e948d5aa7024c0f3a644c4"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "c839efa65e9577576f69116c616a719d"
server_key_iv: "31268a15b90b9fd3d7023dd80589e299"
server_key_version: "encryptionkey@0"
I20260924 07:21:13.463016 19773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:21:13.474337 19773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:21:13.476189 19773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:21:13.476351 19773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:21:13.476575 19773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:21:13.476882 19773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:21:13.476936 19773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:13.476979 19773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:21:13.477006 19773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:13.482808 19773 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:41901
I20260924 07:21:13.482848 19901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:41901 every 8 connection(s)
I20260924 07:21:13.483161 19773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:21:13.489210 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 19773
Sep 24 07:21:13 dist-test-slave-rc1w krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:21:13.489328 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:21:13.489511 4057 external_mini_cluster.cc:1428] Setting key e213c58c74bf5d7d45433b464b405bb7
I20260924 07:21:13.492697 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.483815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:55453 (local address 127.3.246.126:42205)
0924 07:21:13.483885 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:13.483888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:13.484081 (+ 193us) server_negotiation.cc:407] Connection header received
0924 07:21:13.484631 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:13.484633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:13.484648 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:13.484724 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:21:13.485728 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.486241 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.487263 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.487468 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.487559 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.489865 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:21:13.489866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:21:13.489867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:21:13.489875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:21:13.491481 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:13.491895 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:13.491895 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:13.491896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:13.491957 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:13.492199 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:13.492200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:13.492200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:13.492314 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:21:13.492366 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:13.492564 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 07:21:13.492632 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:21:13.492944 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.483784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:55453)
0924 07:21:13.483939 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:13.483986 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:13.484511 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:13.484801 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:13.484805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:13.485382 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:21:13.485570 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.485575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.486363 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.486364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.487168 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.487171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.487571 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.487572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.487699 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.488237 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:21:13.488248 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:21:13.489779 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:21:13.491561 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:13.491567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:13.491569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:13.491800 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:13.492043 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:13.492043 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:13.492044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:13.492141 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:13.492355 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:21:13.492358 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:21:13.492470 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 07:21:13.492550 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:21:13.492670 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260924 07:21:13.493340 19902 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42205
I20260924 07:21:13.493425 19902 heartbeater.cc:464] Registering TS with master...
I20260924 07:21:13.493577 19902 heartbeater.cc:511] Master 127.3.246.126:42205 requested a full tablet report, sending...
I20260924 07:21:13.493975 19443 ts_manager.cc:194] Registered new tserver with Master: 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901)
I20260924 07:21:13.494364 19443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:55453
I20260924 07:21:13.501434 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:21:13.509016 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.503030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.3.246.126:42205)
0924 07:21:13.503095 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:13.503096 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:13.503123 (+ 27us) server_negotiation.cc:407] Connection header received
0924 07:21:13.503179 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:13.503180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:13.503189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:13.503237 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:21:13.503756 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.504256 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.504990 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.505222 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.505278 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.506518 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:21:13.506520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:21:13.506521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:21:13.506527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:21:13.507940 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:13.508260 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:13.508261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:13.508261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:13.508313 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:13.508483 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:13.508483 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:13.508483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:13.508628 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:21:13.508705 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:13.508873 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 07:21:13.508942 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:21:13.510854 19443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:21:13.512488 19443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:21:13.518957 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.515076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:41901 (local address 127.0.0.1:49146)
0924 07:21:13.515342 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:13.515383 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:13.515454 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:13.515698 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:13.515702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:13.515704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:21:13.515778 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.515781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.516807 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.516808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.517871 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.517874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.518661 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.518662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.518750 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.518766 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 07:21:13.518833 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:21:13.518884 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":56,"threads_started":1}
I20260924 07:21:13.519369 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.515240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.3.246.67:41901)
0924 07:21:13.515443 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:13.515448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:13.515462 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:21:13.515562 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:13.515565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:13.515583 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:13.515860 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:21:13.515940 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.516695 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.517976 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.518563 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.519147 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.519222 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:13.519249 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 07:21:13.519289 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":96,"threads_started":1}
I20260924 07:21:13.519739 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.515077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:42187 (local address 127.0.0.1:53864)
0924 07:21:13.515690 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:13.515699 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:13.515740 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:13.516029 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:13.516030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:13.516033 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:21:13.516067 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.516068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.517375 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.517376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.518495 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.518498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.519501 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.519503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.519565 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.519570 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:21:13.519621 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:21:13.519664 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260924 07:21:13.519905 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.515077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:36999 (local address 127.0.0.1:52368)
0924 07:21:13.515505 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:13.515515 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:13.515583 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:13.515874 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:13.515875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:13.515877 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:21:13.515934 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.515936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.517228 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.517229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.518705 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.518707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.519673 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.519674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.519746 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.519750 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:21:13.519801 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:21:13.519838 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":95,"threads_started":1}
I20260924 07:21:13.520052 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.515262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.3.246.65:42187)
0924 07:21:13.515457 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:13.515461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:13.515802 (+ 341us) server_negotiation.cc:407] Connection header received
0924 07:21:13.515850 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:13.515853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:13.515868 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:13.515949 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:21:13.516158 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.517227 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.518584 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.519340 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.519407 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.519464 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:13.519921 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 07:21:13.519965 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260924 07:21:13.520202 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.515342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52368 (local address 127.3.246.66:36999)
0924 07:21:13.515625 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:13.515644 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:13.515654 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:21:13.515694 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:13.515696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:13.515712 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:13.515788 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:21:13.515995 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.517113 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.518785 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.519499 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.519577 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.519633 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:13.520067 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 07:21:13.520115 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":93,"threads_started":1}
I20260924 07:21:13.520143 19836 tablet_service.cc:1467] Processing CreateTablet for tablet 37fbd093e53e42609fbe2c56ced9f493 (DEFAULT_TABLE table=test-table [id=6a79e0357e644c9caaae22908bf6673e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:21:13.520529 19836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37fbd093e53e42609fbe2c56ced9f493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:21:13.520735 19568 tablet_service.cc:1467] Processing CreateTablet for tablet 37fbd093e53e42609fbe2c56ced9f493 (DEFAULT_TABLE table=test-table [id=6a79e0357e644c9caaae22908bf6673e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:21:13.520888 19702 tablet_service.cc:1467] Processing CreateTablet for tablet 37fbd093e53e42609fbe2c56ced9f493 (DEFAULT_TABLE table=test-table [id=6a79e0357e644c9caaae22908bf6673e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:21:13.521067 19568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37fbd093e53e42609fbe2c56ced9f493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:21:13.521200 19702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37fbd093e53e42609fbe2c56ced9f493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:21:13.523355 19919 tablet_bootstrap.cc:492] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4: Bootstrap starting.
I20260924 07:21:13.523955 19919 tablet_bootstrap.cc:654] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4: Neither blocks nor log segments found. Creating new log.
I20260924 07:21:13.524317 19921 tablet_bootstrap.cc:492] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6: Bootstrap starting.
I20260924 07:21:13.524471 19919 log.cc:824] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:21:13.525043 19921 tablet_bootstrap.cc:654] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6: Neither blocks nor log segments found. Creating new log.
I20260924 07:21:13.525326 19919 tablet_bootstrap.cc:492] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4: No bootstrap required, opened a new log
I20260924 07:21:13.525410 19919 ts_tablet_manager.cc:1397] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:21:13.525581 19921 log.cc:824] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:21:13.525835 19920 tablet_bootstrap.cc:492] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Bootstrap starting.
I20260924 07:21:13.526364 19921 tablet_bootstrap.cc:492] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6: No bootstrap required, opened a new log
I20260924 07:21:13.526436 19921 ts_tablet_manager.cc:1397] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:21:13.526487 19920 tablet_bootstrap.cc:654] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Neither blocks nor log segments found. Creating new log.
I20260924 07:21:13.526921 19920 log.cc:824] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Log is configured to *not* fsync() on all Append() calls
I20260924 07:21:13.527557 19920 tablet_bootstrap.cc:492] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: No bootstrap required, opened a new log
I20260924 07:21:13.527431 19919 raft_consensus.cc:348] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } }
I20260924 07:21:13.527616 19919 raft_consensus.cc:374] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:21:13.527637 19920 ts_tablet_manager.cc:1397] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:21:13.527691 19919 raft_consensus.cc:729] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887681cd91e948d5aa7024c0f3a644c4, State: Initialized, Role: FOLLOWER
I20260924 07:21:13.527824 19919 consensus_queue.cc:260] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } }
I20260924 07:21:13.528095 19919 ts_tablet_manager.cc:1428] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:21:13.528219 19902 heartbeater.cc:503] Master 127.3.246.126:42205 was elected leader, sending a full tablet report...
I20260924 07:21:13.528757 19921 raft_consensus.cc:348] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } }
I20260924 07:21:13.528972 19921 raft_consensus.cc:374] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:21:13.529052 19921 raft_consensus.cc:729] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e9282915e448b1aa06900beb8f61d6, State: Initialized, Role: FOLLOWER
I20260924 07:21:13.529171 19921 consensus_queue.cc:260] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } }
I20260924 07:21:13.529452 19921 ts_tablet_manager.cc:1428] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:21:13.529594 19768 heartbeater.cc:503] Master 127.3.246.126:42205 was elected leader, sending a full tablet report...
I20260924 07:21:13.529657 19920 raft_consensus.cc:348] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } }
I20260924 07:21:13.529865 19920 raft_consensus.cc:374] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:21:13.529922 19920 raft_consensus.cc:729] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94efd2fe59c246109c6b64ff2411d533, State: Initialized, Role: FOLLOWER
I20260924 07:21:13.530037 19920 consensus_queue.cc:260] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } }
I20260924 07:21:13.530300 19920 ts_tablet_manager.cc:1428] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:21:13.530344 19634 heartbeater.cc:503] Master 127.3.246.126:42205 was elected leader, sending a full tablet report...
W20260924 07:21:13.577237 19769 tablet.cc:2369] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:21:13.671960 19635 tablet.cc:2369] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:21:13.719950 19926 raft_consensus.cc:482] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:21:13.720082 19926 raft_consensus.cc:504] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } }
I20260924 07:21:13.720577 19926 leader_election.cc:288] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94efd2fe59c246109c6b64ff2411d533 (127.3.246.65:42187), 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901)
I20260924 07:21:13.723805 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.720889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:41689 (local address 127.3.246.65:42187)
0924 07:21:13.720944 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:13.720946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:13.720957 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:21:13.720991 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:13.720992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:13.721004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:13.721117 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:21:13.721286 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.722080 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.723097 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.723462 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.723520 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.723527 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:13.723690 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 07:21:13.723734 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 07:21:13.723806 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.720772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:42187 (local address 127.3.246.66:41689)
0924 07:21:13.720827 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:13.720840 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:13.720892 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:13.721124 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:13.721125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:13.721127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:21:13.721207 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.721209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.722160 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.722161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.723006 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.723009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.723538 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.723539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.723618 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.723622 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:21:13.723670 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 07:21:13.723721 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":2}
I20260924 07:21:13.723850 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.720897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:48497 (local address 127.3.246.67:41901)
0924 07:21:13.720983 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:13.720984 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:13.721037 (+ 53us) server_negotiation.cc:407] Connection header received
0924 07:21:13.721127 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:13.721128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:13.721137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:13.721197 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:21:13.721371 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.721983 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.723098 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.723487 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.723578 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.723586 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:13.723734 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 07:21:13.723772 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:21:13.723881 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.720787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:41901 (local address 127.3.246.66:48497)
0924 07:21:13.720977 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:13.720987 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:13.721052 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:13.721209 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:13.721210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:13.721211 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:21:13.721294 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.721296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.722045 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.722046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.723006 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:13.723009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.723592 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.723592 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:13.723645 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.723648 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:21:13.723719 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:21:13.723756 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":29,"thread_start_us":128,"threads_started":1}
I20260924 07:21:13.724273 19856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fbd093e53e42609fbe2c56ced9f493" candidate_uuid: "a0e9282915e448b1aa06900beb8f61d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887681cd91e948d5aa7024c0f3a644c4" is_pre_election: true
I20260924 07:21:13.724349 19588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fbd093e53e42609fbe2c56ced9f493" candidate_uuid: "a0e9282915e448b1aa06900beb8f61d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94efd2fe59c246109c6b64ff2411d533" is_pre_election: true
I20260924 07:21:13.724439 19856 raft_consensus.cc:2454] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e9282915e448b1aa06900beb8f61d6 in term 0.
I20260924 07:21:13.724504 19588 raft_consensus.cc:2454] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e9282915e448b1aa06900beb8f61d6 in term 0.
I20260924 07:21:13.724669 19657 leader_election.cc:302] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 887681cd91e948d5aa7024c0f3a644c4, a0e9282915e448b1aa06900beb8f61d6; no voters:
I20260924 07:21:13.724828 19926 raft_consensus.cc:2781] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:21:13.724874 19926 raft_consensus.cc:482] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:21:13.724890 19926 raft_consensus.cc:3037] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:21:13.725548 19926 raft_consensus.cc:504] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } }
I20260924 07:21:13.725709 19926 leader_election.cc:288] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [CANDIDATE]: Term 1 election: Requested vote from peers 94efd2fe59c246109c6b64ff2411d533 (127.3.246.65:42187), 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901)
I20260924 07:21:13.725860 19588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fbd093e53e42609fbe2c56ced9f493" candidate_uuid: "a0e9282915e448b1aa06900beb8f61d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94efd2fe59c246109c6b64ff2411d533"
I20260924 07:21:13.725945 19588 raft_consensus.cc:3037] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:21:13.725960 19856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fbd093e53e42609fbe2c56ced9f493" candidate_uuid: "a0e9282915e448b1aa06900beb8f61d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887681cd91e948d5aa7024c0f3a644c4"
I20260924 07:21:13.726027 19856 raft_consensus.cc:3037] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:21:13.726828 19588 raft_consensus.cc:2454] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e9282915e448b1aa06900beb8f61d6 in term 1.
I20260924 07:21:13.726828 19856 raft_consensus.cc:2454] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e9282915e448b1aa06900beb8f61d6 in term 1.
I20260924 07:21:13.727018 19655 leader_election.cc:302] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94efd2fe59c246109c6b64ff2411d533, a0e9282915e448b1aa06900beb8f61d6; no voters:
I20260924 07:21:13.727116 19926 raft_consensus.cc:2781] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:21:13.727262 19926 raft_consensus.cc:686] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [term 1 LEADER]: Becoming Leader. State: Replica: a0e9282915e448b1aa06900beb8f61d6, State: Running, Role: LEADER
I20260924 07:21:13.727357 19926 consensus_queue.cc:237] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } }
I20260924 07:21:13.728197 19443 catalog_manager.cc:5495] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66). New cstate: current_term: 1 leader_uuid: "a0e9282915e448b1aa06900beb8f61d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } health_report { overall_health: HEALTHY } } }
W20260924 07:21:13.733721 19903 tablet.cc:2369] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:21:13.747087 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.744689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.3.246.66:36999)
0924 07:21:13.744754 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:13.744756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:13.744793 (+ 37us) server_negotiation.cc:407] Connection header received
0924 07:21:13.744828 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:13.744829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:13.744845 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:13.744898 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:21:13.745043 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.745501 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.746349 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:13.746538 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:13.746590 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:13.746736 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:21:13.746833 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:21:13.746892 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:13.746932 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 07:21:13.746951 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:21:13.749046 19856 raft_consensus.cc:1261] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4 [term 1 FOLLOWER]: Refusing update from remote peer a0e9282915e448b1aa06900beb8f61d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:21:13.749046 19588 raft_consensus.cc:1261] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Refusing update from remote peer a0e9282915e448b1aa06900beb8f61d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:21:13.749459 19926 consensus_queue.cc:1035] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:21:13.749620 19929 consensus_queue.cc:1035] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:21:13.752123 19933 mvcc.cc:205] Tried to move back new op lower bound from 7332800404472938496 to 7332800404387782656. Current Snapshot: MvccSnapshot[applied={T|T < 7332800404472938496}]
I20260924 07:21:13.752617 19932 mvcc.cc:205] Tried to move back new op lower bound from 7332800404472938496 to 7332800404387782656. Current Snapshot: MvccSnapshot[applied={T|T < 7332800404472938496}]
I20260924 07:21:13.754645 19443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:21:13.754755 19443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:21:13.755601 19443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Sending DeleteTablet for 3 replicas of tablet 37fbd093e53e42609fbe2c56ced9f493
I20260924 07:21:13.755959 19836 tablet_service.cc:1514] Processing DeleteTablet for tablet 37fbd093e53e42609fbe2c56ced9f493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:21:13 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:49146
I20260924 07:21:13.755959 19568 tablet_service.cc:1514] Processing DeleteTablet for tablet 37fbd093e53e42609fbe2c56ced9f493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:21:13 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:53864
I20260924 07:21:13.755959 19702 tablet_service.cc:1514] Processing DeleteTablet for tablet 37fbd093e53e42609fbe2c56ced9f493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:21:13 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:52368
I20260924 07:21:13.756193 19944 tablet_replica.cc:330] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6: stopping tablet replica
I20260924 07:21:13.756287 19943 tablet_replica.cc:330] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: stopping tablet replica
I20260924 07:21:13.756282 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 19505
I20260924 07:21:13.756361 19944 raft_consensus.cc:2229] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:21:13.756508 19944 raft_consensus.cc:2258] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:21:13.758030 19944 ts_tablet_manager.cc:1905] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:21:13.758455 19942 tablet_replica.cc:330] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4: stopping tablet replica
I20260924 07:21:13.758575 19942 raft_consensus.cc:2229] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:21:13.758675 19942 raft_consensus.cc:2258] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:21:13.759097 19942 ts_tablet_manager.cc:1905] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:21:13.759959 19944 ts_tablet_manager.cc:1918] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:21:13.760095 19944 log.cc:1196] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/37fbd093e53e42609fbe2c56ced9f493
I20260924 07:21:13.760484 19944 ts_tablet_manager.cc:1939] T 37fbd093e53e42609fbe2c56ced9f493 P a0e9282915e448b1aa06900beb8f61d6: Deleting consensus metadata
I20260924 07:21:13.760964 19428 catalog_manager.cc:4841] TS a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999): tablet 37fbd093e53e42609fbe2c56ced9f493 (table test-table [id=6a79e0357e644c9caaae22908bf6673e]) successfully deleted
I20260924 07:21:13.761109 19942 ts_tablet_manager.cc:1918] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:21:13.761185 19942 log.cc:1196] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/37fbd093e53e42609fbe2c56ced9f493
I20260924 07:21:13.761502 19942 ts_tablet_manager.cc:1939] T 37fbd093e53e42609fbe2c56ced9f493 P 887681cd91e948d5aa7024c0f3a644c4: Deleting consensus metadata
I20260924 07:21:13.761998 19429 catalog_manager.cc:4841] TS 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901): tablet 37fbd093e53e42609fbe2c56ced9f493 (table test-table [id=6a79e0357e644c9caaae22908bf6673e]) successfully deleted
W20260924 07:21:13.762759 19427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:42187 (error 108)
I20260924 07:21:13.763332 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:13.763193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:42187 (local address 127.0.0.1:53866)
0924 07:21:13.763258 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:13.763300 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:42187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 07:21:13.763386 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 19639
W20260924 07:21:13.765848 19427 catalog_manager.cc:4584] TS 94efd2fe59c246109c6b64ff2411d533 (127.3.246.65:42187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37fbd093e53e42609fbe2c56ced9f493: Network error: Client connection negotiation failed: client connection to 127.3.246.65:42187: connect: Connection refused (error 111)
I20260924 07:21:13.769450 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 19773
I20260924 07:21:13.774449 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 19411
I20260924 07:21:33.779752 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:42205
--webserver_interface=127.3.246.126
--webserver_port=34191
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:42205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:21:33.862408 19946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:21:33.862666 19946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:21:33.862703 19946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:21:33.863843 19946 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:21:33.863914 19946 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:21:33.863935 19946 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:21:33.863961 19946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:21:33.863992 19946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:21:33.864017 19946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:21:33.864049 19946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:21:33.864080 19946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:21:33.865542 19946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:42205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:42205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=34191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:21:33.865846 19946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:21:33.866082 19946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:21:33.874503 19953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:33.874498 19955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:33.874493 19952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:33.874653 19946 server_base.cc:1034] running on GCE node
I20260924 07:21:33.875103 19946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:21:33.875463 19946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:21:33.876784 19946 hybrid_clock.cc:648] HybridClock initialized: now 1790234493876776 us; error 64 us; skew 500 ppm
Sep 24 07:21:33 dist-test-slave-rc1w krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234493, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:21:33.879417 19946 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:21:33.879902 19946 webserver.cc:460] Webserver started at http://127.3.246.126:34191/ using document root <none> and password file <none>
I20260924 07:21:33.880110 19946 fs_manager.cc:359] Metadata directory not provided
I20260924 07:21:33.880162 19946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:21:33.881462 19946 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:21:33.882213 19962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:33.882426 19946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:21:33.882550 19946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "e430d6416d454b74b9981d6def6d33a8"
format_stamp: "Formatted at 2026-09-24 07:21:12 on dist-test-slave-rc1w"
server_key: "dff6069f8e31002db52513ae325abb50"
server_key_iv: "15098e55d39bf04561ceb85625d0b8a5"
server_key_version: "encryptionkey@0"
I20260924 07:21:33.882956 19946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:21:33.892773 19946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:21:33.894824 19946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:21:33.894982 19946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:21:33.899192 19946 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:42205
I20260924 07:21:33.899224 20014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:42205 every 8 connection(s)
I20260924 07:21:33.899618 19946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:21:33.901080 20015 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:21:33.901867 20015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Bootstrap starting.
I20260924 07:21:33.905544 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 19946
I20260924 07:21:33.905893 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:42187
--local_ip_for_outbound_sockets=127.3.246.65
--tserver_master_addrs=127.3.246.126:42205
--webserver_port=39737
--webserver_interface=127.3.246.65
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:21:33.910854 20015 log.cc:824] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Log is configured to *not* fsync() on all Append() calls
I20260924 07:21:33.912791 20015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:21:33.913120 20015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Bootstrap complete.
I20260924 07:21:33.915478 20015 raft_consensus.cc:348] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } }
I20260924 07:21:33.915724 20015 raft_consensus.cc:729] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e430d6416d454b74b9981d6def6d33a8, State: Initialized, Role: FOLLOWER
I20260924 07:21:33.915922 20015 consensus_queue.cc:260] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } }
I20260924 07:21:33.915987 20015 raft_consensus.cc:388] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:21:33.916057 20015 raft_consensus.cc:482] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:21:33.916106 20015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:21:33.917142 20015 raft_consensus.cc:504] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } }
I20260924 07:21:33.917254 20015 leader_election.cc:302] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e430d6416d454b74b9981d6def6d33a8; no voters:
I20260924 07:21:33.917466 20015 leader_election.cc:288] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:21:33.917500 20019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:21:33.917613 20019 raft_consensus.cc:686] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 2 LEADER]: Becoming Leader. State: Replica: e430d6416d454b74b9981d6def6d33a8, State: Running, Role: LEADER
I20260924 07:21:33.917768 20015 sys_catalog.cc:564] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:21:33.917784 20019 consensus_queue.cc:237] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } }
I20260924 07:21:33.918076 20019 sys_catalog.cc:455] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e430d6416d454b74b9981d6def6d33a8. Latest consensus state: current_term: 2 leader_uuid: "e430d6416d454b74b9981d6def6d33a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } } }
I20260924 07:21:33.918147 20019 sys_catalog.cc:458] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: This master's current role is: LEADER
I20260924 07:21:33.918255 20020 sys_catalog.cc:455] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e430d6416d454b74b9981d6def6d33a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } } }
I20260924 07:21:33.918386 20020 sys_catalog.cc:458] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: This master's current role is: LEADER
I20260924 07:21:33.918443 20024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:21:33.919330 20024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:21:33.919582 20024 catalog_manager.cc:1259] Loaded cluster ID: 41efdacd3a2a417684807662d661e034
I20260924 07:21:33.919770 20024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:21:33.920933 20024 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:21:33.921097 20024 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Loaded TSK: 0
I20260924 07:21:33.929456 20024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Generated new TSK 1
I20260924 07:21:33.929562 20024 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:21:33.989600 20017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:21:33.989890 20017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:21:33.989933 20017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:21:33.991297 20017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:21:33.991370 20017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:21:33.991417 20017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:21:33.993018 20017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:42187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=39737
--tserver_master_addrs=127.3.246.126:42205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:21:33.993331 20017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:21:33.993594 20017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:21:33.998868 20045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:33.998999 20043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:33.999157 20017 server_base.cc:1034] running on GCE node
W20260924 07:21:33.998868 20042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:33.999418 20017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:21:33.999725 20017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:21:34.000919 20017 hybrid_clock.cc:648] HybridClock initialized: now 1790234494000897 us; error 45 us; skew 500 ppm
Sep 24 07:21:33 dist-test-slave-rc1w krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234493, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:21:34.004139 20017 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:21:34.004716 20017 webserver.cc:460] Webserver started at http://127.3.246.65:39737/ using document root <none> and password file <none>
I20260924 07:21:34.004989 20017 fs_manager.cc:359] Metadata directory not provided
I20260924 07:21:34.005076 20017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:21:34.006991 20017 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:21:34.007808 20052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:34.008042 20017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:21:34.008142 20017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "94efd2fe59c246109c6b64ff2411d533"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "2fba192a80780e2cd4083be9840dca9d"
server_key_iv: "20d4b1757b16e9e6d3cc01cec1fb0c47"
server_key_version: "encryptionkey@0"
I20260924 07:21:34.008626 20017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:21:34.037909 20017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:21:34.040068 20017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:21:34.040241 20017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:21:34.040482 20017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:21:34.040943 20059 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:21:34.042922 20017 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:21:34.042981 20017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 07:21:34.043025 20017 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:21:34.043668 20017 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:21:34.043722 20017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 07:21:34.043803 20059 tablet_bootstrap.cc:492] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Bootstrap starting.
I20260924 07:21:34.050980 20017 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:42187
I20260924 07:21:34.051443 20017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:21:34.053844 20166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:42187 every 8 connection(s)
I20260924 07:21:34.054719 20059 log.cc:824] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Log is configured to *not* fsync() on all Append() calls
I20260924 07:21:34.057057 20059 tablet_bootstrap.cc:492] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:21:34.057359 20059 tablet_bootstrap.cc:492] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Bootstrap complete.
I20260924 07:21:34.058599 20059 ts_tablet_manager.cc:1397] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260924 07:21:34.059747 20059 raft_consensus.cc:348] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } }
I20260924 07:21:34.059968 20059 raft_consensus.cc:729] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94efd2fe59c246109c6b64ff2411d533, State: Initialized, Role: FOLLOWER
I20260924 07:21:34.060101 20059 consensus_queue.cc:260] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } }
I20260924 07:21:34.060369 20059 ts_tablet_manager.cc:1428] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260924 07:21:34.061595 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 20017
I20260924 07:21:34.061919 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:36999
--local_ip_for_outbound_sockets=127.3.246.66
--tserver_master_addrs=127.3.246.126:42205
--webserver_port=39453
--webserver_interface=127.3.246.66
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:21:34 dist-test-slave-rc1w krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234493, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:21:34.067270 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.052368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:40651 (local address 127.3.246.126:42205)
0924 07:21:34.056181 (+ 3813us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:34.056192 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:34.056241 (+ 49us) server_negotiation.cc:407] Connection header received
0924 07:21:34.056402 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:34.056418 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:34.056475 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:34.056617 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:21:34.057509 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.058985 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.060299 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.060770 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.060902 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.063668 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:21:34.063672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:21:34.063672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:21:34.063681 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:21:34.065324 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:34.065885 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:34.065887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:34.065888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:34.065975 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:34.066289 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:34.066289 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:34.066290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:34.066493 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:21:34.066566 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:34.067023 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 07:21:34.067133 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3748,"thread_start_us":78,"threads_started":1}
I20260924 07:21:34.067348 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.052236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:40651)
0924 07:21:34.055018 (+ 2782us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:34.055062 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:34.055728 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:34.056683 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:34.056687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:34.057054 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:21:34.057336 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.057343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.059105 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.059106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.060211 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.060215 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.060879 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.060880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.061011 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.061640 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:21:34.061647 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:21:34.063555 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:21:34.065431 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:34.065438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:34.065440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:34.065800 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:34.066055 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:34.066056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:34.066056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:34.066213 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:34.066577 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:21:34.066584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:21:34.066761 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 07:21:34.066882 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 07:21:34.067089 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2740,"thread_start_us":112,"threads_started":1}
I20260924 07:21:34.068476 20167 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42205
I20260924 07:21:34.068572 20167 heartbeater.cc:464] Registering TS with master...
I20260924 07:21:34.068742 20167 heartbeater.cc:511] Master 127.3.246.126:42205 requested a full tablet report, sending...
I20260924 07:21:34.069458 19979 ts_manager.cc:194] Registered new tserver with Master: 94efd2fe59c246109c6b64ff2411d533 (127.3.246.65:42187)
I20260924 07:21:34.070645 19979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:40651
I20260924 07:21:34.071458 20167 heartbeater.cc:503] Master 127.3.246.126:42205 was elected leader, sending a full tablet report...
Sep 24 07:21:34 dist-test-slave-rc1w krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234493, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for kudu/127.3.246.65@KRBTEST.COM
I20260924 07:21:34.079017 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.070205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.3.246.65:42187)
0924 07:21:34.070360 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:34.070365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:34.070539 (+ 174us) server_negotiation.cc:407] Connection header received
0924 07:21:34.070638 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:34.070640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:34.070655 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:34.070774 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:21:34.071534 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.072500 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.073602 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.073955 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.074014 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.076102 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:21:34.076110 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:21:34.076111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:21:34.076118 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:21:34.077729 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:34.078148 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:34.078151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:34.078151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:34.078209 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:34.078409 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:34.078410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:34.078410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:34.078559 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:21:34.078648 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:34.078840 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 07:21:34.078930 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260924 07:21:34.079021 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.070104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:42187 (local address 127.0.0.1:48850)
0924 07:21:34.070413 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:34.070459 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:34.070546 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:34.070985 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:34.070988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:34.071363 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:21:34.071429 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.071433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.072584 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.072586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.073517 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.073519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.074027 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.074029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.074108 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.074556 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:21:34.074562 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:21:34.075992 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:21:34.077830 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:34.077837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:34.077839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:34.078052 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:34.078278 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:34.078279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:34.078279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:34.078344 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:34.078639 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:21:34.078646 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:21:34.078696 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 07:21:34.078790 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 07:21:34.078912 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":90,"threads_started":1}
I20260924 07:21:34.079402 20101 tablet_service.cc:1514] Processing DeleteTablet for tablet 37fbd093e53e42609fbe2c56ced9f493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:21:13 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:48850
I20260924 07:21:34.079926 20176 tablet_replica.cc:330] stopping tablet replica
I20260924 07:21:34.080037 20176 raft_consensus.cc:2229] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:21:34.080104 20176 raft_consensus.cc:2258] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:21:34.080410 20176 ts_tablet_manager.cc:1905] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:21:34.082805 20176 ts_tablet_manager.cc:1918] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:21:34.082883 20176 log.cc:1196] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal/wals/37fbd093e53e42609fbe2c56ced9f493
I20260924 07:21:34.083113 20176 ts_tablet_manager.cc:1939] T 37fbd093e53e42609fbe2c56ced9f493 P 94efd2fe59c246109c6b64ff2411d533: Deleting consensus metadata
I20260924 07:21:34.083666 19963 catalog_manager.cc:4841] TS 94efd2fe59c246109c6b64ff2411d533 (127.3.246.65:42187): tablet 37fbd093e53e42609fbe2c56ced9f493 (table test-table [id=6a79e0357e644c9caaae22908bf6673e]) successfully deleted
W20260924 07:21:34.147605 20173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:21:34.147814 20173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:21:34.147837 20173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:21:34.148923 20173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:21:34.148958 20173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:21:34.148984 20173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:21:34.150424 20173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:36999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=39453
--tserver_master_addrs=127.3.246.126:42205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:21:34.150604 20173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:21:34.150780 20173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:21:34.155820 20184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:34.155833 20181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:34.155833 20182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:34.156378 20173 server_base.cc:1034] running on GCE node
I20260924 07:21:34.156566 20173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:21:34.156756 20173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:21:34.157898 20173 hybrid_clock.cc:648] HybridClock initialized: now 1790234494157868 us; error 41 us; skew 500 ppm
Sep 24 07:21:34 dist-test-slave-rc1w krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234494, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:21:34.160506 20173 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:21:34.161015 20173 webserver.cc:460] Webserver started at http://127.3.246.66:39453/ using document root <none> and password file <none>
I20260924 07:21:34.161245 20173 fs_manager.cc:359] Metadata directory not provided
I20260924 07:21:34.161330 20173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:21:34.162796 20173 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:21:34.163487 20191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:34.163694 20173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:21:34.163796 20173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "a0e9282915e448b1aa06900beb8f61d6"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "a3cb4c01aa85201b916706371249c356"
server_key_iv: "8e68d6373ff72a6d064d1ffe7360d772"
server_key_version: "encryptionkey@0"
I20260924 07:21:34.164168 20173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:21:34.175107 20173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:21:34.176911 20173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:21:34.177074 20173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:21:34.177340 20173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:21:34.177788 20173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:21:34.177826 20173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:34.177847 20173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:21:34.177877 20173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:34.183712 20173 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:36999
I20260924 07:21:34.183753 20304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:36999 every 8 connection(s)
I20260924 07:21:34.184082 20173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:21:34.187435 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 20173
I20260924 07:21:34.187716 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:41901
--local_ip_for_outbound_sockets=127.3.246.67
--tserver_master_addrs=127.3.246.126:42205
--webserver_port=38177
--webserver_interface=127.3.246.67
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:21:34 dist-test-slave-rc1w krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234494, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:21:34.194125 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.184717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:36889)
0924 07:21:34.184874 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:34.184915 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:34.185437 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:34.185719 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:34.185723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:34.186361 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:21:34.186549 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.186554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.187237 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.187238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.188204 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.188206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.188742 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.188743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.188875 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.189451 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:21:34.189467 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:21:34.190859 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:21:34.192769 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:34.192776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:34.192777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:34.192983 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:34.193199 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:34.193200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:34.193200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:34.193291 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:34.193580 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:21:34.193584 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:21:34.193748 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 07:21:34.193835 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:21:34.193951 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260924 07:21:34.194128 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.184758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:36889 (local address 127.3.246.126:42205)
0924 07:21:34.184804 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:34.184806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:34.185005 (+ 199us) server_negotiation.cc:407] Connection header received
0924 07:21:34.185533 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:34.185534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:34.185544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:34.185622 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:21:34.186634 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.187158 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.188296 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.188656 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.188730 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.190953 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:21:34.190955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:21:34.190956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:21:34.190963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:21:34.192676 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:34.193058 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:34.193060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:34.193061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:34.193126 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:34.193358 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:34.193359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:34.193360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:34.193511 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:21:34.193590 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:34.193881 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 07:21:34.193979 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:21:34.194615 20305 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42205
I20260924 07:21:34.194722 20305 heartbeater.cc:464] Registering TS with master...
I20260924 07:21:34.194928 20305 heartbeater.cc:511] Master 127.3.246.126:42205 requested a full tablet report, sending...
I20260924 07:21:34.195362 19979 ts_manager.cc:194] Registered new tserver with Master: a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999)
I20260924 07:21:34.195756 19979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:36889
W20260924 07:21:34.271515 20308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:21:34.271704 20308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:21:34.271727 20308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:21:34.272877 20308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:21:34.272917 20308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:21:34.272943 20308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:21:34.274428 20308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:41901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=38177
--tserver_master_addrs=127.3.246.126:42205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:21:34.274673 20308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:21:34.274897 20308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:21:34.279906 20314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:34.279928 20316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:34.279918 20313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:34.280164 20308 server_base.cc:1034] running on GCE node
I20260924 07:21:34.280440 20308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:21:34.280684 20308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:21:34.281811 20308 hybrid_clock.cc:648] HybridClock initialized: now 1790234494281798 us; error 36 us; skew 500 ppm
Sep 24 07:21:34 dist-test-slave-rc1w krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234494, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:21:34.284276 20308 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:21:34.284641 20308 webserver.cc:460] Webserver started at http://127.3.246.67:38177/ using document root <none> and password file <none>
I20260924 07:21:34.284842 20308 fs_manager.cc:359] Metadata directory not provided
I20260924 07:21:34.284894 20308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:21:34.286316 20308 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:21:34.286892 20323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:34.287086 20308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:21:34.287176 20308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "887681cd91e948d5aa7024c0f3a644c4"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "c839efa65e9577576f69116c616a719d"
server_key_iv: "31268a15b90b9fd3d7023dd80589e299"
server_key_version: "encryptionkey@0"
I20260924 07:21:34.287483 20308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:21:34.300930 20308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:21:34.303506 20308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:21:34.303648 20308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:21:34.303884 20308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:21:34.304191 20308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:21:34.304224 20308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:34.304265 20308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:21:34.304292 20308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:34.310024 20308 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:41901
I20260924 07:21:34.310101 20436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:41901 every 8 connection(s)
I20260924 07:21:34.310415 20308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:21:34.312819 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 20308
Sep 24 07:21:34 dist-test-slave-rc1w krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234494, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:21:34.320649 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.313259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.3.246.126:42205)
0924 07:21:34.313561 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:34.313562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:34.313568 (+ 6us) server_negotiation.cc:407] Connection header received
0924 07:21:34.313583 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:34.313584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:34.313613 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:34.313688 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:21:34.314466 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.314911 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.315971 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.316151 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.316203 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.317595 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:21:34.317596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:21:34.317596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:21:34.317602 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:21:34.319452 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:34.319840 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:34.319841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:34.319841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:34.319895 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:34.320112 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:34.320113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:34.320113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:34.320248 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:21:34.320417 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:34.320456 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 07:21:34.320539 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":83,"threads_started":1}
I20260924 07:21:34.320784 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.311143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:37487 (local address 127.3.246.126:42205)
0924 07:21:34.311219 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:34.311221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:34.311422 (+ 201us) server_negotiation.cc:407] Connection header received
0924 07:21:34.311880 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:34.311881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:34.311893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:34.311978 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:21:34.313047 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.313755 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.315008 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.315201 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.315250 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.317458 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:21:34.317459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:21:34.317459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:21:34.317466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:21:34.319192 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:34.319655 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:34.319657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:34.319657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:34.319711 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:34.320028 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:34.320029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:34.320029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:34.320162 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:21:34.320210 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:34.320533 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 07:21:34.320611 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:21:34.320869 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.311136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:37487)
0924 07:21:34.311300 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:34.311335 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:34.311798 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:34.312059 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:34.312062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:34.312747 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:21:34.312933 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.312938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.313846 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.313847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.314909 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.314912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.315278 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.315279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.315423 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.315904 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:21:34.315914 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:21:34.317360 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:21:34.319273 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:34.319280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:34.319282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:34.319577 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:34.319783 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:34.319784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:34.319784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:34.319959 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:34.320226 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:21:34.320232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:21:34.320379 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 07:21:34.320493 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 07:21:34.320642 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260924 07:21:34.321480 20437 heartbeater.cc:347] Connected to a master server at 127.3.246.126:42205
I20260924 07:21:34.321592 20437 heartbeater.cc:464] Registering TS with master...
I20260924 07:21:34.321786 20437 heartbeater.cc:511] Master 127.3.246.126:42205 requested a full tablet report, sending...
I20260924 07:21:34.322227 19978 ts_manager.cc:194] Registered new tserver with Master: 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901)
I20260924 07:21:34.322593 19978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:37487
I20260924 07:21:34.332352 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:21:34.335022 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.332710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42560 (local address 127.3.246.126:42205)
0924 07:21:34.332767 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:34.332769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:34.332893 (+ 124us) server_negotiation.cc:407] Connection header received
0924 07:21:34.332943 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:34.332944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:34.332985 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:34.333045 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:21:34.333201 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.333651 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.334446 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.334635 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.334693 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.334819 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:21:34.334856 (+ 37us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 07:21:34.334979 (+ 123us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42560: token expired
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 07:21:34.335099 20441 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42560: token expired
W20260924 07:21:34.336058 19907 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.246.126:42205)
I20260924 07:21:34.343607 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.336323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.3.246.126:42205)
0924 07:21:34.336390 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:34.336392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:34.336401 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:21:34.336422 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:34.336423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:34.336433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:34.336496 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:21:34.336962 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.337724 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.338952 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.339124 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.339179 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.340735 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:21:34.340737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:21:34.340737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:21:34.340745 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:21:34.342506 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:34.342825 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:34.342826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:34.342827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:34.342876 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:34.343099 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:34.343100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:34.343100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:34.343261 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:21:34.343347 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:34.343475 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 07:21:34.343546 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:21:34.344717 19978 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:21:34.346478 19978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:21:34.348685 20101 tablet_service.cc:1467] Processing CreateTablet for tablet 834c0afbddbd414abd84bc5439e26b4f (DEFAULT_TABLE table=test-table [id=cb6c7bfb9cec47d6bb565bd469711cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:21:34.348891 20101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 834c0afbddbd414abd84bc5439e26b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:21:34.350219 20059 tablet_bootstrap.cc:492] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533: Bootstrap starting.
I20260924 07:21:34.350567 20059 tablet_bootstrap.cc:654] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533: Neither blocks nor log segments found. Creating new log.
I20260924 07:21:34.351147 20059 tablet_bootstrap.cc:492] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533: No bootstrap required, opened a new log
I20260924 07:21:34.351202 20059 ts_tablet_manager.cc:1397] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:21:34.351581 20059 raft_consensus.cc:348] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:21:34.351679 20059 raft_consensus.cc:374] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:21:34.351735 20059 raft_consensus.cc:729] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94efd2fe59c246109c6b64ff2411d533, State: Initialized, Role: FOLLOWER
I20260924 07:21:34.351788 20059 consensus_queue.cc:260] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:21:34.351883 20059 ts_tablet_manager.cc:1428] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:21:34.352608 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.348429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:41901 (local address 127.0.0.1:57548)
0924 07:21:34.348685 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:34.348693 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:34.348729 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:34.349093 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:34.349094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:34.349096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:21:34.349141 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.349143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.350062 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.350063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.351173 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.351176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.352351 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.352352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.352410 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.352423 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 07:21:34.352485 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 07:21:34.352530 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":100,"threads_started":1}
I20260924 07:21:34.352643 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.348726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.3.246.67:41901)
0924 07:21:34.348902 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:34.348907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:34.348925 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:21:34.348976 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:34.348978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:34.348995 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:34.349077 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:21:34.349199 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.349980 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.351279 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.352193 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.352257 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.352319 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:34.352521 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 07:21:34.352566 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260924 07:21:34.353415 20371 tablet_service.cc:1467] Processing CreateTablet for tablet 834c0afbddbd414abd84bc5439e26b4f (DEFAULT_TABLE table=test-table [id=cb6c7bfb9cec47d6bb565bd469711cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:21:34.353781 20371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 834c0afbddbd414abd84bc5439e26b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:21:34.353978 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.348417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:36999 (local address 127.0.0.1:58266)
0924 07:21:34.348551 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:34.348562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:34.348600 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:34.350584 (+ 1984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:34.350585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:34.350586 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:21:34.350632 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.350634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.351791 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.351792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.352759 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.352762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.353749 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.353750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.353823 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.353827 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:21:34.353878 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:21:34.353920 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 07:21:34.354027 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.348517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.3.246.66:36999)
0924 07:21:34.350345 (+ 1828us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:34.350349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:34.350360 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:21:34.350402 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:34.350404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:34.350420 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:34.350709 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:21:34.350806 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.351620 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.352863 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.353606 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.353749 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.353803 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:34.353909 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 07:21:34.353948 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1791,"thread_start_us":107,"threads_started":1}
I20260924 07:21:34.356578 20447 tablet_bootstrap.cc:492] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4: Bootstrap starting.
I20260924 07:21:34.357194 20447 tablet_bootstrap.cc:654] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4: Neither blocks nor log segments found. Creating new log.
I20260924 07:21:34.357128 20239 tablet_service.cc:1467] Processing CreateTablet for tablet 834c0afbddbd414abd84bc5439e26b4f (DEFAULT_TABLE table=test-table [id=cb6c7bfb9cec47d6bb565bd469711cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:21:34.357426 20239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 834c0afbddbd414abd84bc5439e26b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:21:34.357807 20447 log.cc:824] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:21:34.358537 20447 tablet_bootstrap.cc:492] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4: No bootstrap required, opened a new log
I20260924 07:21:34.358621 20447 ts_tablet_manager.cc:1397] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:21:34.359916 20447 raft_consensus.cc:348] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:21:34.360080 20447 raft_consensus.cc:374] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:21:34.360127 20447 raft_consensus.cc:729] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887681cd91e948d5aa7024c0f3a644c4, State: Initialized, Role: FOLLOWER
I20260924 07:21:34.360249 20447 consensus_queue.cc:260] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:21:34.360365 20449 tablet_bootstrap.cc:492] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6: Bootstrap starting.
I20260924 07:21:34.360541 20447 ts_tablet_manager.cc:1428] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:21:34.360580 20437 heartbeater.cc:503] Master 127.3.246.126:42205 was elected leader, sending a full tablet report...
I20260924 07:21:34.360992 20449 tablet_bootstrap.cc:654] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6: Neither blocks nor log segments found. Creating new log.
I20260924 07:21:34.361577 20449 log.cc:824] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:21:34.362277 20449 tablet_bootstrap.cc:492] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6: No bootstrap required, opened a new log
I20260924 07:21:34.362357 20449 ts_tablet_manager.cc:1397] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:21:34.363720 20449 raft_consensus.cc:348] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:21:34.363897 20449 raft_consensus.cc:374] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:21:34.363943 20449 raft_consensus.cc:729] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e9282915e448b1aa06900beb8f61d6, State: Initialized, Role: FOLLOWER
I20260924 07:21:34.364076 20449 consensus_queue.cc:260] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:21:34.364363 20449 ts_tablet_manager.cc:1428] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:21:34.364378 20305 heartbeater.cc:503] Master 127.3.246.126:42205 was elected leader, sending a full tablet report...
I20260924 07:21:34.364569 20172 raft_consensus.cc:482] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:21:34.364662 20172 raft_consensus.cc:504] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:21:34.365022 20172 leader_election.cc:288] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999), 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901)
I20260924 07:21:34.368041 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.365224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:43889 (local address 127.3.246.66:36999)
0924 07:21:34.365279 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:34.365281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:34.365357 (+ 76us) server_negotiation.cc:407] Connection header received
0924 07:21:34.365425 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:34.365425 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:34.365435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:34.365486 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:21:34.365637 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.366261 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.367300 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.367644 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.367700 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.367707 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:34.367934 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 07:21:34.367973 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:21:34.368095 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.365155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:41901 (local address 127.3.246.65:49505)
0924 07:21:34.365202 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:34.365212 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:34.365258 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:34.365429 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:34.365430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:34.365431 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:21:34.365495 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.365496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.366284 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.366285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.367162 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.367165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.367813 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.367814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.367897 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.367901 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:21:34.367954 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:21:34.367989 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 07:21:34.368148 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.365169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:36999 (local address 127.3.246.65:43889)
0924 07:21:34.365303 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:34.365312 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:34.365360 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:34.365487 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:34.365488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:34.365489 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:21:34.365547 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.365549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.366361 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.366362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.367212 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:34.367215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.367742 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.367743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:34.367821 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.367843 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:21:34.367917 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:21:34.367962 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260924 07:21:34.368263 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.365235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:49505 (local address 127.3.246.67:41901)
0924 07:21:34.365309 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:34.365311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:34.365319 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:21:34.365340 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:34.365340 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:34.365349 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:34.365411 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:21:34.365577 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.366204 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.367227 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.367729 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.367800 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.367808 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:34.368171 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 07:21:34.368209 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:21:34.368444 20259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834c0afbddbd414abd84bc5439e26b4f" candidate_uuid: "94efd2fe59c246109c6b64ff2411d533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e9282915e448b1aa06900beb8f61d6" is_pre_election: true
I20260924 07:21:34.368579 20259 raft_consensus.cc:2454] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94efd2fe59c246109c6b64ff2411d533 in term 0.
I20260924 07:21:34.368587 20391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834c0afbddbd414abd84bc5439e26b4f" candidate_uuid: "94efd2fe59c246109c6b64ff2411d533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887681cd91e948d5aa7024c0f3a644c4" is_pre_election: true
I20260924 07:21:34.368716 20391 raft_consensus.cc:2454] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94efd2fe59c246109c6b64ff2411d533 in term 0.
I20260924 07:21:34.368750 20054 leader_election.cc:302] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94efd2fe59c246109c6b64ff2411d533, a0e9282915e448b1aa06900beb8f61d6; no voters:
I20260924 07:21:34.368878 20172 raft_consensus.cc:2781] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:21:34.368927 20172 raft_consensus.cc:482] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:21:34.368955 20172 raft_consensus.cc:3037] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:21:34.369566 20172 raft_consensus.cc:504] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:21:34.369732 20172 leader_election.cc:288] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 1 election: Requested vote from peers a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999), 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901)
I20260924 07:21:34.369858 20259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834c0afbddbd414abd84bc5439e26b4f" candidate_uuid: "94efd2fe59c246109c6b64ff2411d533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e9282915e448b1aa06900beb8f61d6"
I20260924 07:21:34.369866 20391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834c0afbddbd414abd84bc5439e26b4f" candidate_uuid: "94efd2fe59c246109c6b64ff2411d533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887681cd91e948d5aa7024c0f3a644c4"
I20260924 07:21:34.369946 20259 raft_consensus.cc:3037] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:21:34.369943 20391 raft_consensus.cc:3037] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:21:34.370560 20259 raft_consensus.cc:2454] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94efd2fe59c246109c6b64ff2411d533 in term 1.
I20260924 07:21:34.370615 20391 raft_consensus.cc:2454] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94efd2fe59c246109c6b64ff2411d533 in term 1.
I20260924 07:21:34.370751 20054 leader_election.cc:302] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94efd2fe59c246109c6b64ff2411d533, a0e9282915e448b1aa06900beb8f61d6; no voters:
I20260924 07:21:34.370877 20172 raft_consensus.cc:2781] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:21:34.371021 20172 raft_consensus.cc:686] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 LEADER]: Becoming Leader. State: Replica: 94efd2fe59c246109c6b64ff2411d533, State: Running, Role: LEADER
I20260924 07:21:34.371106 20172 consensus_queue.cc:237] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:21:34.371831 19978 catalog_manager.cc:5495] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94efd2fe59c246109c6b64ff2411d533 (127.3.246.65). New cstate: current_term: 1 leader_uuid: "94efd2fe59c246109c6b64ff2411d533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } health_report { overall_health: HEALTHY } } }
I20260924 07:21:34.381312 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.379231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.3.246.65:42187)
0924 07:21:34.379268 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:34.379270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:34.379279 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:21:34.379336 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:34.379337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:34.379351 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:34.379411 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:21:34.379517 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.379986 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.380668 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:34.380884 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:34.380942 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:34.381067 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:21:34.381152 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:21:34.381208 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:34.381247 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 07:21:34.381264 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:21:34.382694 20259 raft_consensus.cc:1261] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6 [term 1 FOLLOWER]: Refusing update from remote peer 94efd2fe59c246109c6b64ff2411d533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:21:34.382694 20391 raft_consensus.cc:1261] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4 [term 1 FOLLOWER]: Refusing update from remote peer 94efd2fe59c246109c6b64ff2411d533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:21:34.383029 20172 consensus_queue.cc:1035] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:21:34.383164 20454 consensus_queue.cc:1035] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:21:34.385843 20457 mvcc.cc:205] Tried to move back new op lower bound from 7332800488989360128 to 7332800488944676864. Current Snapshot: MvccSnapshot[applied={T|T < 7332800488989360128 or (T in {7332800488989360128})}]
I20260924 07:21:34.387193 19978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:21:34.387281 19978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:21:34.388331 19978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Sending DeleteTablet for 3 replicas of tablet 834c0afbddbd414abd84bc5439e26b4f
I20260924 07:21:34.388551 20371 tablet_service.cc:1514] Processing DeleteTablet for tablet 834c0afbddbd414abd84bc5439e26b4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:21:34 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:57548
I20260924 07:21:34.388684 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 20017
I20260924 07:21:34.388754 20239 tablet_service.cc:1514] Processing DeleteTablet for tablet 834c0afbddbd414abd84bc5439e26b4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:21:34 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:58266
I20260924 07:21:34.388811 20467 tablet_replica.cc:330] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4: stopping tablet replica
I20260924 07:21:34.389004 20467 raft_consensus.cc:2229] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:21:34.389153 20467 raft_consensus.cc:2258] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:21:34.389380 20371 tablet_service.cc:1514] Processing DeleteTablet for tablet 834c0afbddbd414abd84bc5439e26b4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:21:34 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:57548
I20260924 07:21:34.389631 20468 tablet_replica.cc:330] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6: stopping tablet replica
I20260924 07:21:34.389747 20468 raft_consensus.cc:2229] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:21:34.389750 20467 ts_tablet_manager.cc:1905] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:21:34.389849 20468 raft_consensus.cc:2258] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:21:34.390269 20468 ts_tablet_manager.cc:1905] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:21:34.391636 20467 ts_tablet_manager.cc:1918] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:21:34.391706 20467 log.cc:1196] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/834c0afbddbd414abd84bc5439e26b4f
I20260924 07:21:34.392022 20467 ts_tablet_manager.cc:1939] T 834c0afbddbd414abd84bc5439e26b4f P 887681cd91e948d5aa7024c0f3a644c4: Deleting consensus metadata
I20260924 07:21:34.392480 19965 catalog_manager.cc:4841] TS 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901): tablet 834c0afbddbd414abd84bc5439e26b4f (table test-table [id=cb6c7bfb9cec47d6bb565bd469711cbb]) successfully deleted
W20260924 07:21:34.392731 19965 catalog_manager.cc:4808] TS 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901): delete failed for tablet 834c0afbddbd414abd84bc5439e26b4f because the tablet was not found. No further retry: Not found: Tablet not found: 834c0afbddbd414abd84bc5439e26b4f
I20260924 07:21:34.393946 20468 ts_tablet_manager.cc:1918] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:21:34.394023 20468 log.cc:1196] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/834c0afbddbd414abd84bc5439e26b4f
I20260924 07:21:34.394289 20468 ts_tablet_manager.cc:1939] T 834c0afbddbd414abd84bc5439e26b4f P a0e9282915e448b1aa06900beb8f61d6: Deleting consensus metadata
I20260924 07:21:34.394677 19964 catalog_manager.cc:4841] TS a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999): tablet 834c0afbddbd414abd84bc5439e26b4f (table test-table [id=cb6c7bfb9cec47d6bb565bd469711cbb]) successfully deleted
W20260924 07:21:34.396351 19963 connection.cc:535] client connection to 127.3.246.65:42187 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 07:21:34.396447 19963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:21:34.396842 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 20173
I20260924 07:21:34.397012 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:34.396871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:42187 (local address 127.0.0.1:48862)
0924 07:21:34.396939 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:34.396978 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:42187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 07:21:34.398115 19963 catalog_manager.cc:4584] TS 94efd2fe59c246109c6b64ff2411d533 (127.3.246.65:42187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 834c0afbddbd414abd84bc5439e26b4f: Network error: Client connection negotiation failed: client connection to 127.3.246.65:42187: connect: Connection refused (error 111)
I20260924 07:21:34.401647 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 20308
I20260924 07:21:34.406435 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 19946
I20260924 07:21:54.411917 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:42205
--webserver_interface=127.3.246.126
--webserver_port=34191
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:42205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:21:54.493125 20470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:21:54.493352 20470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:21:54.493391 20470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:21:54.494642 20470 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:21:54.494710 20470 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:21:54.494735 20470 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:21:54.494763 20470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:21:54.494791 20470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:21:54.494820 20470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:21:54.494844 20470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:21:54.494906 20470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:21:54.496405 20470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:42205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:42205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=34191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:21:54.496678 20470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:21:54.496918 20470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:21:54.503170 20476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:54.503252 20475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:54.503175 20478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:54.503443 20470 server_base.cc:1034] running on GCE node
I20260924 07:21:54.503731 20470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:21:54.504081 20470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:21:54.505240 20470 hybrid_clock.cc:648] HybridClock initialized: now 1790234514505235 us; error 31 us; skew 500 ppm
Sep 24 07:21:54 dist-test-slave-rc1w krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234514, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:21:54.507642 20470 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:21:54.508105 20470 webserver.cc:460] Webserver started at http://127.3.246.126:34191/ using document root <none> and password file <none>
I20260924 07:21:54.508339 20470 fs_manager.cc:359] Metadata directory not provided
I20260924 07:21:54.508409 20470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:21:54.509817 20470 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:21:54.510578 20485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:54.510778 20470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:21:54.510877 20470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "e430d6416d454b74b9981d6def6d33a8"
format_stamp: "Formatted at 2026-09-24 07:21:12 on dist-test-slave-rc1w"
server_key: "dff6069f8e31002db52513ae325abb50"
server_key_iv: "15098e55d39bf04561ceb85625d0b8a5"
server_key_version: "encryptionkey@0"
I20260924 07:21:54.511234 20470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:21:54.524267 20470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:21:54.526085 20470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:21:54.526242 20470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:21:54.530192 20470 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:42205
I20260924 07:21:54.530223 20537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:42205 every 8 connection(s)
I20260924 07:21:54.530582 20470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:21:54.531955 20538 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:21:54.532749 20538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Bootstrap starting.
I20260924 07:21:54.537079 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 20470
I20260924 07:21:54.537428 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:42187
--local_ip_for_outbound_sockets=127.3.246.65
--tserver_master_addrs=127.3.246.126:42205
--webserver_port=39737
--webserver_interface=127.3.246.65
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:21:54.541736 20538 log.cc:824] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Log is configured to *not* fsync() on all Append() calls
I20260924 07:21:54.543937 20538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:21:54.544286 20538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Bootstrap complete.
I20260924 07:21:54.546553 20538 raft_consensus.cc:348] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } }
I20260924 07:21:54.546778 20538 raft_consensus.cc:729] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e430d6416d454b74b9981d6def6d33a8, State: Initialized, Role: FOLLOWER
I20260924 07:21:54.546933 20538 consensus_queue.cc:260] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } }
I20260924 07:21:54.547026 20538 raft_consensus.cc:388] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:21:54.547079 20538 raft_consensus.cc:482] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:21:54.547152 20538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:21:54.548146 20538 raft_consensus.cc:504] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } }
I20260924 07:21:54.548261 20538 leader_election.cc:302] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e430d6416d454b74b9981d6def6d33a8; no voters:
I20260924 07:21:54.548431 20538 leader_election.cc:288] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 07:21:54.548487 20542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 07:21:54.548640 20542 raft_consensus.cc:686] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [term 3 LEADER]: Becoming Leader. State: Replica: e430d6416d454b74b9981d6def6d33a8, State: Running, Role: LEADER
I20260924 07:21:54.548704 20538 sys_catalog.cc:564] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:21:54.548781 20542 consensus_queue.cc:237] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } }
I20260924 07:21:54.549073 20542 sys_catalog.cc:455] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e430d6416d454b74b9981d6def6d33a8. Latest consensus state: current_term: 3 leader_uuid: "e430d6416d454b74b9981d6def6d33a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } } }
I20260924 07:21:54.549147 20542 sys_catalog.cc:458] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: This master's current role is: LEADER
I20260924 07:21:54.549075 20543 sys_catalog.cc:455] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e430d6416d454b74b9981d6def6d33a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e430d6416d454b74b9981d6def6d33a8" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 42205 } } }
I20260924 07:21:54.549367 20543 sys_catalog.cc:458] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8 [sys.catalog]: This master's current role is: LEADER
I20260924 07:21:54.549317 20548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:21:54.550259 20548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:21:54.550417 20548 catalog_manager.cc:1259] Loaded cluster ID: 41efdacd3a2a417684807662d661e034
I20260924 07:21:54.550595 20548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:21:54.551770 20548 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:21:54.551944 20548 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Loaded TSK: 0
I20260924 07:21:54.551991 20548 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e430d6416d454b74b9981d6def6d33a8: Loaded TSK: 1
F20260924 07:21:54.552211 20548 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa0787a49ad google::LogMessage::Fail() at ??:0
@ 0x7fa0787a788a google::LogMessage::SendToLog() at ??:0
@ 0x7fa0787a44de google::LogMessage::Flush() at ??:0
@ 0x7fa0787a5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa07af11104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa07af0ed41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa0806fe1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa0806fe48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa0806fe594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fa0806c4c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa0806cead0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa07a6720d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa07a667954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa07b4cb6db start_thread at ??:0
@ 0x7fa0772a261f clone at ??:0
W20260924 07:21:54.649185 20540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:21:54.649470 20540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:21:54.649509 20540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:21:54.651187 20540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:21:54.651252 20540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:21:54.651293 20540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:21:54.653362 20540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:42187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=39737
--tserver_master_addrs=127.3.246.126:42205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:21:54.653628 20540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:21:54.653931 20540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:21:54.659329 20582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:54.659343 20580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:54.659333 20579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:54.659750 20540 server_base.cc:1034] running on GCE node
I20260924 07:21:54.659945 20540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:21:54.660223 20540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:21:54.661389 20540 hybrid_clock.cc:648] HybridClock initialized: now 1790234514661376 us; error 33 us; skew 500 ppm
Sep 24 07:21:54 dist-test-slave-rc1w krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234514, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:21:54.663882 20540 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:21:54.664307 20540 webserver.cc:460] Webserver started at http://127.3.246.65:39737/ using document root <none> and password file <none>
I20260924 07:21:54.664547 20540 fs_manager.cc:359] Metadata directory not provided
I20260924 07:21:54.664615 20540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:21:54.666090 20540 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:21:54.666872 20589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:54.667078 20540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:21:54.667188 20540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "94efd2fe59c246109c6b64ff2411d533"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "2fba192a80780e2cd4083be9840dca9d"
server_key_iv: "20d4b1757b16e9e6d3cc01cec1fb0c47"
server_key_version: "encryptionkey@0"
I20260924 07:21:54.667529 20540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:21:54.679035 20540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:21:54.681288 20540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:21:54.681444 20540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:21:54.681711 20540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:21:54.682152 20596 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:21:54.684185 20540 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:21:54.684242 20540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 07:21:54.684283 20540 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:21:54.684906 20540 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:21:54.684959 20540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 07:21:54.685029 20596 tablet_bootstrap.cc:492] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533: Bootstrap starting.
I20260924 07:21:54.692512 20540 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:42187
I20260924 07:21:54.692617 20703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:42187 every 8 connection(s)
I20260924 07:21:54.692996 20540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:21:54.694070 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 20540
I20260924 07:21:54.694430 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:36999
--local_ip_for_outbound_sockets=127.3.246.66
--tserver_master_addrs=127.3.246.126:42205
--webserver_port=39453
--webserver_interface=127.3.246.66
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:21:54.696584 20596 log.cc:824] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533: Log is configured to *not* fsync() on all Append() calls
I20260924 07:21:54.699496 20596 tablet_bootstrap.cc:492] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:21:54.699842 20596 tablet_bootstrap.cc:492] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533: Bootstrap complete.
I20260924 07:21:54.701046 20596 ts_tablet_manager.cc:1397] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.005s
I20260924 07:21:54.702718 20596 raft_consensus.cc:348] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:21:54.703068 20596 raft_consensus.cc:729] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94efd2fe59c246109c6b64ff2411d533, State: Initialized, Role: FOLLOWER
I20260924 07:21:54.703229 20596 consensus_queue.cc:260] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:21:54.703524 20596 ts_tablet_manager.cc:1428] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533: Time spent starting tablet: real 0.002s user 0.003s sys 0.003s
W20260924 07:21:54.764549 20706 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:21:54.693786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:49717)
0924 07:21:54.694245 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:54.694292 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:54.695223 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:54.764350 (+ 69127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":92,"threads_started":1}
W20260924 07:21:54.764824 20704 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:42205 (0 consecutive failures): Network error: Failed to ping master at 127.3.246.126:42205: Client connection negotiation failed: client connection to 127.3.246.126:42205: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:21:54.765196 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:54.765060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:48859)
0924 07:21:54.765119 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:54.765171 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 07:21:54.765578 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:54.765476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:53451)
0924 07:21:54.765526 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:54.765556 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 07:21:54.765722 20704 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:21:54.777061 20707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:21:54.777249 20707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:21:54.777271 20707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:21:54.778481 20707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:21:54.778524 20707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:21:54.778550 20707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:21:54.779958 20707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:36999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=39453
--tserver_master_addrs=127.3.246.126:42205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:21:54.780197 20707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:21:54.780426 20707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:21:54.785300 20715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:54.785445 20707 server_base.cc:1034] running on GCE node
W20260924 07:21:54.785296 20717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:54.785342 20714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:54.785781 20707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:21:54.786026 20707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:21:54.787184 20707 hybrid_clock.cc:648] HybridClock initialized: now 1790234514787173 us; error 26 us; skew 500 ppm
Sep 24 07:21:54 dist-test-slave-rc1w krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234514, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:21:54.789649 20707 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:21:54.790102 20707 webserver.cc:460] Webserver started at http://127.3.246.66:39453/ using document root <none> and password file <none>
I20260924 07:21:54.790294 20707 fs_manager.cc:359] Metadata directory not provided
I20260924 07:21:54.790335 20707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:21:54.791530 20707 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:21:54.792104 20724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:54.792289 20707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:21:54.792375 20707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "a0e9282915e448b1aa06900beb8f61d6"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "a3cb4c01aa85201b916706371249c356"
server_key_iv: "8e68d6373ff72a6d064d1ffe7360d772"
server_key_version: "encryptionkey@0"
I20260924 07:21:54.792673 20707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:21:54.811172 20707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:21:54.812901 20707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:21:54.813009 20707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:21:54.813179 20707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:21:54.813488 20707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:21:54.813521 20707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:54.813544 20707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:21:54.813603 20707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:54.819155 20707 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:36999
I20260924 07:21:54.819232 20837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:36999 every 8 connection(s)
I20260924 07:21:54.819579 20707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:21:54.820019 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 20707
I20260924 07:21:54.820348 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:41901
--local_ip_for_outbound_sockets=127.3.246.67
--tserver_master_addrs=127.3.246.126:42205
--webserver_port=38177
--webserver_interface=127.3.246.67
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:21:54.820676 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:54.820286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:50885)
0924 07:21:54.820448 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:54.820526 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
W20260924 07:21:54.821043 20838 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:42205 (0 consecutive failures): Network error: Failed to ping master at 127.3.246.126:42205: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
I20260924 07:21:54.821352 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:54.821249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:59907)
0924 07:21:54.821301 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:54.821325 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 07:21:54.821621 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:54.821535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:54281)
0924 07:21:54.821579 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:54.821601 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 07:21:54.821760 20838 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:21:54.900058 20841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:21:54.900254 20841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:21:54.900276 20841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:21:54.901338 20841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:21:54.901372 20841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:21:54.901430 20841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:21:54.903014 20841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:41901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=38177
--tserver_master_addrs=127.3.246.126:42205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:21:54.903195 20841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:21:54.903370 20841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:21:54.908442 20846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:54.908486 20841 server_base.cc:1034] running on GCE node
W20260924 07:21:54.908452 20847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:21:54.908411 20849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:21:54.908762 20841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:21:54.909030 20841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:21:54.910199 20841 hybrid_clock.cc:648] HybridClock initialized: now 1790234514910177 us; error 37 us; skew 500 ppm
Sep 24 07:21:54 dist-test-slave-rc1w krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234514, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:21:54.912578 20841 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:21:54.913013 20841 webserver.cc:460] Webserver started at http://127.3.246.67:38177/ using document root <none> and password file <none>
I20260924 07:21:54.913280 20841 fs_manager.cc:359] Metadata directory not provided
I20260924 07:21:54.913347 20841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:21:54.914803 20841 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:21:54.915450 20856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:54.915632 20841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:21:54.915742 20841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "887681cd91e948d5aa7024c0f3a644c4"
format_stamp: "Formatted at 2026-09-24 07:21:13 on dist-test-slave-rc1w"
server_key: "c839efa65e9577576f69116c616a719d"
server_key_iv: "31268a15b90b9fd3d7023dd80589e299"
server_key_version: "encryptionkey@0"
I20260924 07:21:54.916116 20841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:21:54.940692 20841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:21:54.942724 20841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:21:54.942888 20841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:21:54.943138 20841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:21:54.943447 20841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:21:54.943512 20841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:54.943558 20841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:21:54.943581 20841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:21:54.949536 20841 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:41901
I20260924 07:21:54.949596 20969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:41901 every 8 connection(s)
I20260924 07:21:54.949977 20841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:21:54.951097 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:54.950715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:57997)
0924 07:21:54.950887 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:54.950961 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
W20260924 07:21:54.951509 20970 heartbeater.cc:644] Failed to heartbeat to 127.3.246.126:42205 (0 consecutive failures): Network error: Failed to ping master at 127.3.246.126:42205: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
I20260924 07:21:54.951821 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:54.951704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:48293)
0924 07:21:54.951770 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:54.951801 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 07:21:54.952226 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:54.952132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:36015)
0924 07:21:54.952178 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:54.952209 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 07:21:54.952330 20970 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:21:54.955427 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 20841
I20260924 07:21:55.766402 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:55.766135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:43139)
0924 07:21:55.766292 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:55.766337 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260924 07:21:55.822290 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:55.822072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:39843)
0924 07:21:55.822208 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:55.822244 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
I20260924 07:21:55.952927 20976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:55.952665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:41933)
0924 07:21:55.952819 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:55.952861 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":57,"threads_started":1}
I20260924 07:21:56.330498 20977 raft_consensus.cc:482] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:21:56.330628 20977 raft_consensus.cc:504] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:21:56.331103 20977 leader_election.cc:288] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999), 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901)
Sep 24 07:21:56 dist-test-slave-rc1w krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234514, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.66@KRBTEST.COM
Sep 24 07:21:56 dist-test-slave-rc1w krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234514, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.67@KRBTEST.COM
I20260924 07:21:56.357765 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:56.331305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:41901 (local address 127.3.246.65:48681)
0924 07:21:56.331416 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:56.331428 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:56.331487 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:56.332363 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:56.332366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:56.332775 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:21:56.332967 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:56.332973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:56.334127 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:56.334128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:56.335091 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:56.335093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:56.335836 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:56.335837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:56.335913 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:56.351543 (+ 15630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:21:56.351557 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:21:56.353571 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:21:56.355672 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:56.355673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:56.355673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:56.356064 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:56.356422 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:56.356423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:56.356423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:56.356576 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:56.357074 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:21:56.357081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:21:56.357259 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 07:21:56.357399 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 07:21:56.357611 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":42,"thread_start_us":48,"threads_started":1}
I20260924 07:21:56.357772 20981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:56.331522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:46675 (local address 127.3.246.66:36999)
0924 07:21:56.331637 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:56.331646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:56.331674 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:21:56.332110 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:56.332112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:56.332154 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:56.332330 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:21:56.333113 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:56.334067 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:56.335229 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:56.335675 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:56.335762 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:56.353404 (+ 17642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:21:56.353410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:21:56.353410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:21:56.353438 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:21:56.355492 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:56.356195 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:56.356199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:56.356199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:56.356339 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:56.356654 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:56.356655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:56.356655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:56.356994 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:21:56.357085 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:56.357491 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 07:21:56.357638 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":42,"threads_started":1}
I20260924 07:21:56.357786 20980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:56.331460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:48681 (local address 127.3.246.67:41901)
0924 07:21:56.331631 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 07:21:56.331641 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:21:56.331673 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:21:56.332110 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:21:56.332112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:21:56.332154 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:21:56.332292 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:21:56.333100 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:56.334035 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:56.335298 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:21:56.335738 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:21:56.335858 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:56.353729 (+ 17871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:21:56.353735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:21:56.353736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:21:56.353748 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:21:56.355583 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:56.356196 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:56.356199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:56.356199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:56.356339 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:21:56.356655 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:21:56.356656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:21:56.356657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:21:56.356994 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:21:56.357085 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:21:56.357491 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 07:21:56.357638 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260924 07:21:56.357776 20979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:56.331305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:36999 (local address 127.3.246.65:46675)
0924 07:21:56.331415 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:56.331428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:21:56.331487 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:21:56.332364 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:21:56.332367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:21:56.332772 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:21:56.332967 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:56.332973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:56.334214 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:56.334215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:56.335065 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:21:56.335068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:21:56.335780 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:21:56.335782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:21:56.335907 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:21:56.351540 (+ 15633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:21:56.351551 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:21:56.353239 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:21:56.355593 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:56.355600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:56.355602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:56.356064 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:56.356422 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:21:56.356423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:21:56.356423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:21:56.356576 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:21:56.357074 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:21:56.357081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:21:56.357259 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 07:21:56.357398 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 07:21:56.357668 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":185,"thread_start_us":59,"threads_started":1}
W20260924 07:21:56.358902 20591 leader_election.cc:341] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999): Not found: Tablet not found: 834c0afbddbd414abd84bc5439e26b4f
W20260924 07:21:56.358979 20592 leader_election.cc:341] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901): Not found: Tablet not found: 834c0afbddbd414abd84bc5439e26b4f
I20260924 07:21:56.359105 20592 leader_election.cc:302] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94efd2fe59c246109c6b64ff2411d533; no voters: 887681cd91e948d5aa7024c0f3a644c4, a0e9282915e448b1aa06900beb8f61d6
I20260924 07:21:56.359236 20977 raft_consensus.cc:2726] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:21:56.767138 20979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:56.766968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:47753)
0924 07:21:56.767046 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:56.767088 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 07:21:56.822958 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:56.822764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:37737)
0924 07:21:56.822879 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:56.822917 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260924 07:21:56.953615 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:56.953415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:50437)
0924 07:21:56.953544 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:56.953580 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260924 07:21:57.767916 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:57.767668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:39741)
0924 07:21:57.767833 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:57.767870 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260924 07:21:57.823586 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:57.823404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:56719)
0924 07:21:57.823515 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:57.823549 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260924 07:21:57.954607 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:57.954145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:39219)
0924 07:21:57.954479 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:57.954551 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":283,"threads_started":1}
I20260924 07:21:58.342522 20987 raft_consensus.cc:482] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:21:58.342615 20987 raft_consensus.cc:504] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:21:58.342777 20987 leader_election.cc:288] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999), 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901)
W20260924 07:21:58.343173 20591 leader_election.cc:341] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999): Not found: Tablet not found: 834c0afbddbd414abd84bc5439e26b4f
W20260924 07:21:58.343246 20592 leader_election.cc:341] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901): Not found: Tablet not found: 834c0afbddbd414abd84bc5439e26b4f
I20260924 07:21:58.343318 20592 leader_election.cc:302] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94efd2fe59c246109c6b64ff2411d533; no voters: 887681cd91e948d5aa7024c0f3a644c4, a0e9282915e448b1aa06900beb8f61d6
I20260924 07:21:58.343398 20987 raft_consensus.cc:2726] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:21:58.768661 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:58.768402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:54369)
0924 07:21:58.768557 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:58.768597 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260924 07:21:58.824393 20989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:58.824100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:57039)
0924 07:21:58.824259 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:58.824298 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
I20260924 07:21:58.955291 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:58.955086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:34643)
0924 07:21:58.955219 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:58.955255 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260924 07:21:59.769505 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:59.769241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:48755)
0924 07:21:59.769381 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:59.769447 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260924 07:21:59.825094 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:59.824881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:43505)
0924 07:21:59.825022 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:59.825056 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":57,"threads_started":1}
I20260924 07:21:59.955952 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:21:59.955748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:45629)
0924 07:21:59.955874 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 07:21:59.955913 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260924 07:22:00.770306 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:00.770030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:37255)
0924 07:22:00.770204 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:00.770256 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":66,"threads_started":1}
I20260924 07:22:00.825985 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:00.825608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:53407)
0924 07:22:00.825889 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:00.825941 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":89,"threads_started":1}
I20260924 07:22:00.956736 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:00.956485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:42473)
0924 07:22:00.956638 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:00.956685 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":60,"threads_started":1}
I20260924 07:22:01.162748 20997 raft_consensus.cc:482] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:22:01.162890 20997 raft_consensus.cc:504] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:22:01.163103 20997 leader_election.cc:288] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999), 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901)
W20260924 07:22:01.163592 20591 leader_election.cc:341] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999): Not found: Tablet not found: 834c0afbddbd414abd84bc5439e26b4f
W20260924 07:22:01.163693 20592 leader_election.cc:341] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901): Not found: Tablet not found: 834c0afbddbd414abd84bc5439e26b4f
I20260924 07:22:01.163792 20592 leader_election.cc:302] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94efd2fe59c246109c6b64ff2411d533; no voters: 887681cd91e948d5aa7024c0f3a644c4, a0e9282915e448b1aa06900beb8f61d6
I20260924 07:22:01.163868 20997 raft_consensus.cc:2726] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:22:01.770979 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:01.770770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:51501)
0924 07:22:01.770897 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:01.770933 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260924 07:22:01.826625 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:01.826420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:55205)
0924 07:22:01.826547 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:01.826585 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260924 07:22:01.957422 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:01.957214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:46831)
0924 07:22:01.957344 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:01.957383 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260924 07:22:02.771714 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:02.771472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:55183)
0924 07:22:02.771611 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:02.771666 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260924 07:22:02.827272 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:02.827051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:51641)
0924 07:22:02.827194 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:02.827229 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260924 07:22:02.958143 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:02.957886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:41413)
0924 07:22:02.958060 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:02.958100 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20260924 07:22:03.772416 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:03.772207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:50197)
0924 07:22:03.772333 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:03.772369 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260924 07:22:03.827943 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:03.827741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:54981)
0924 07:22:03.827863 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:03.827900 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260924 07:22:03.958791 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:03.958595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:38625)
0924 07:22:03.958719 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:03.958753 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260924 07:22:04.340772 21007 raft_consensus.cc:482] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:22:04.340878 21007 raft_consensus.cc:504] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:22:04.341095 21007 leader_election.cc:288] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999), 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901)
W20260924 07:22:04.341557 20591 leader_election.cc:341] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999): Not found: Tablet not found: 834c0afbddbd414abd84bc5439e26b4f
W20260924 07:22:04.341637 20592 leader_election.cc:341] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901): Not found: Tablet not found: 834c0afbddbd414abd84bc5439e26b4f
I20260924 07:22:04.341717 20591 leader_election.cc:302] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94efd2fe59c246109c6b64ff2411d533; no voters: 887681cd91e948d5aa7024c0f3a644c4, a0e9282915e448b1aa06900beb8f61d6
I20260924 07:22:04.341801 21007 raft_consensus.cc:2726] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:22:04.773087 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:04.772864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:48801)
0924 07:22:04.773002 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:04.773038 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260924 07:22:04.828691 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:04.828487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:49531)
0924 07:22:04.828606 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:04.828640 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260924 07:22:04.959565 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:04.959327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:39711)
0924 07:22:04.959468 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:04.959516 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260924 07:22:05.773830 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:05.773587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:59099)
0924 07:22:05.773742 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:05.773782 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260924 07:22:05.829404 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:05.829177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:37607)
0924 07:22:05.829321 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:05.829359 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260924 07:22:05.960299 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:05.960075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:45067)
0924 07:22:05.960208 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:05.960252 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260924 07:22:06.774672 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:06.774422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:47943)
0924 07:22:06.774572 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:06.774621 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":60,"threads_started":1}
I20260924 07:22:06.830125 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:06.829907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:58169)
0924 07:22:06.830037 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:06.830081 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260924 07:22:06.961207 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:06.960913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:39831)
0924 07:22:06.961109 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:06.961162 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":105,"threads_started":1}
I20260924 07:22:07.349732 21017 raft_consensus.cc:482] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:22:07.349828 21017 raft_consensus.cc:504] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e9282915e448b1aa06900beb8f61d6" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 36999 } } peers { permanent_uuid: "887681cd91e948d5aa7024c0f3a644c4" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 41901 } } peers { permanent_uuid: "94efd2fe59c246109c6b64ff2411d533" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 42187 } }
I20260924 07:22:07.349996 21017 leader_election.cc:288] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999), 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901)
W20260924 07:22:07.350387 20592 leader_election.cc:341] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 887681cd91e948d5aa7024c0f3a644c4 (127.3.246.67:41901): Not found: Tablet not found: 834c0afbddbd414abd84bc5439e26b4f
W20260924 07:22:07.350447 20591 leader_election.cc:341] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0e9282915e448b1aa06900beb8f61d6 (127.3.246.66:36999): Not found: Tablet not found: 834c0afbddbd414abd84bc5439e26b4f
I20260924 07:22:07.350478 20591 leader_election.cc:302] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94efd2fe59c246109c6b64ff2411d533; no voters: 887681cd91e948d5aa7024c0f3a644c4, a0e9282915e448b1aa06900beb8f61d6
I20260924 07:22:07.350560 21017 raft_consensus.cc:2726] T 834c0afbddbd414abd84bc5439e26b4f P 94efd2fe59c246109c6b64ff2411d533 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:22:07.775350 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:07.775145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:36823)
0924 07:22:07.775267 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:07.775303 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260924 07:22:07.830892 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:07.830692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:53173)
0924 07:22:07.830822 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:07.830851 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260924 07:22:07.962147 21020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:07.961871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:46735)
0924 07:22:07.962070 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:07.962104 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":127,"threads_started":1}
I20260924 07:22:08.776156 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:08.775890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:59989)
0924 07:22:08.776053 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:08.776103 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
I20260924 07:22:08.831656 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:08.831424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:53821)
0924 07:22:08.831555 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:08.831602 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260924 07:22:08.962956 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:08.962712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:46237)
0924 07:22:08.962864 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:08.962907 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260924 07:22:09.776989 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:09.776757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.65:57651)
0924 07:22:09.776900 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:09.776941 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
I20260924 07:22:09.832324 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:09.832127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.66:44915)
0924 07:22:09.832254 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:09.832283 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260924 07:22:09.956332 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 20540
I20260924 07:22:09.961067 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 20707
I20260924 07:22:09.963786 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:09.963546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:42205 (local address 127.3.246.67:47353)
0924 07:22:09.963694 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:09.963744 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.126:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260924 07:22:09.965605 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 20841
2026-09-24T07:22:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58296 ms)
[----------] 30 tests from SecurityITest (228354 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:10 dist-test-slave-rc1w krb5kdc[21029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:10 dist-test-slave-rc1w krb5kdc[21029](info): set up 2 sockets
Sep 24 07:22:10 dist-test-slave-rc1w krb5kdc[21029](info): commencing operation
krb5kdc: starting...
W20260924 07:22:11.017524 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:11 dist-test-slave-rc1w krb5kdc[21029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:22:11.132151 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:35403
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:35403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:11.210319 21042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:11.210505 21042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:11.210526 21042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:11.211640 21042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:11.211679 21042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:11.211694 21042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:11.211705 21042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:11.211717 21042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:11.213176 21042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:35403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:35403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:11.213415 21042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:11.213640 21042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:11.218710 21048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:11.218747 21042 server_base.cc:1034] running on GCE node
W20260924 07:22:11.218845 21050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:11.218858 21047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:11.219252 21042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:11.219524 21042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:11.220662 21042 hybrid_clock.cc:648] HybridClock initialized: now 1790234531220654 us; error 29 us; skew 500 ppm
Sep 24 07:22:11 dist-test-slave-rc1w krb5kdc[21029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234531, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:11.223075 21042 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:22:11.223531 21042 webserver.cc:460] Webserver started at http://127.3.246.126:45205/ using document root <none> and password file <none>
I20260924 07:22:11.223752 21042 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:11.223796 21042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:11.223948 21042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:11.224961 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "45f906aed782472faf3edd89d9315b05"
format_stamp: "Formatted at 2026-09-24 07:22:11 on dist-test-slave-rc1w"
server_key: "346f70e5d1f03e956d55363089af9f02"
server_key_iv: "bec1e9d4b7dd3f0739fc1b77bba6913c"
server_key_version: "encryptionkey@0"
I20260924 07:22:11.225317 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "45f906aed782472faf3edd89d9315b05"
format_stamp: "Formatted at 2026-09-24 07:22:11 on dist-test-slave-rc1w"
server_key: "346f70e5d1f03e956d55363089af9f02"
server_key_iv: "bec1e9d4b7dd3f0739fc1b77bba6913c"
server_key_version: "encryptionkey@0"
I20260924 07:22:11.226696 21042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:22:11.227380 21057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:11.227555 21042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:11.227638 21042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "45f906aed782472faf3edd89d9315b05"
format_stamp: "Formatted at 2026-09-24 07:22:11 on dist-test-slave-rc1w"
server_key: "346f70e5d1f03e956d55363089af9f02"
server_key_iv: "bec1e9d4b7dd3f0739fc1b77bba6913c"
server_key_version: "encryptionkey@0"
I20260924 07:22:11.227710 21042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:11.237335 21042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:11.239064 21042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:11.239217 21042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:11.243110 21042 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:35403
I20260924 07:22:11.243160 21109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:35403 every 8 connection(s)
I20260924 07:22:11.243484 21042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:11.244287 21110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:11.246199 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 21042
I20260924 07:22:11.246300 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:11.246500 4057 external_mini_cluster.cc:1428] Setting key 1e455acffbda14bf477f1c1aa385b528
I20260924 07:22:11.246745 21110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05: Bootstrap starting.
I20260924 07:22:11.247373 21110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:11.247658 21110 log.cc:824] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:11.248313 21110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05: No bootstrap required, opened a new log
I20260924 07:22:11.250581 21110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f906aed782472faf3edd89d9315b05" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35403 } }
I20260924 07:22:11.250758 21110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:11.250793 21110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f906aed782472faf3edd89d9315b05, State: Initialized, Role: FOLLOWER
I20260924 07:22:11.250902 21110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f906aed782472faf3edd89d9315b05" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35403 } }
I20260924 07:22:11.250970 21110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:11.250999 21110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:11.251042 21110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:11.251741 21110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f906aed782472faf3edd89d9315b05" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35403 } }
I20260924 07:22:11.251853 21110 leader_election.cc:302] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f906aed782472faf3edd89d9315b05; no voters:
I20260924 07:22:11.252022 21110 leader_election.cc:288] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:11.252092 21115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:11.252242 21115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [term 1 LEADER]: Becoming Leader. State: Replica: 45f906aed782472faf3edd89d9315b05, State: Running, Role: LEADER
I20260924 07:22:11.252298 21110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:11.252359 21115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f906aed782472faf3edd89d9315b05" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35403 } }
I20260924 07:22:11.252822 21117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f906aed782472faf3edd89d9315b05. Latest consensus state: current_term: 1 leader_uuid: "45f906aed782472faf3edd89d9315b05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f906aed782472faf3edd89d9315b05" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35403 } } }
I20260924 07:22:11.252899 21117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:11.253050 21116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45f906aed782472faf3edd89d9315b05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f906aed782472faf3edd89d9315b05" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 35403 } } }
I20260924 07:22:11.253101 21116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:11.253504 21122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 07:22:11 dist-test-slave-rc1w krb5kdc[21029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:11.253983 21122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:11.255832 21122 catalog_manager.cc:1347] Generated new cluster ID: 2179da35ef724172a273c40c5bf6d62e
I20260924 07:22:11.255883 21122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:11.262332 21122 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:11.263012 21122 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:11.270185 21122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05: Generated new TSK 0
I20260924 07:22:11.270350 21122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:11.272975 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.246944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49716 (local address 127.3.246.126:35403)
0924 07:22:11.247105 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:11.247115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:11.247171 (+ 56us) server_negotiation.cc:407] Connection header received
0924 07:22:11.247645 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:11.247652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:11.247719 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:11.247874 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:11.248521 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.249909 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.250663 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.251262 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.251375 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.254286 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:11.254290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:11.254291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:11.254302 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:11.271413 (+ 17111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:11.271865 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:11.271868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:11.271868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:11.271952 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:11.272154 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:11.272154 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:11.272155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:11.272351 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:11.272449 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:11.272591 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 07:22:11.272710 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:22:11.297933 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:35403
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:11.379012 21136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:11.379212 21136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:11.379235 21136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:11.380384 21136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:11.380426 21136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:11.380451 21136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:22:11.381918 21136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:35403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:11.382177 21136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:11.382418 21136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:11.387389 21141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:11.387403 21142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:11.387405 21144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:11.387580 21136 server_base.cc:1034] running on GCE node
I20260924 07:22:11.387801 21136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:11.388057 21136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:11.389222 21136 hybrid_clock.cc:648] HybridClock initialized: now 1790234531389210 us; error 29 us; skew 500 ppm
Sep 24 07:22:11 dist-test-slave-rc1w krb5kdc[21029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234531, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:11.397730 21136 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:22:11.398157 21136 webserver.cc:460] Webserver started at http://127.3.246.65:45039/ using document root <none> and password file <none>
I20260924 07:22:11.398373 21136 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:11.398439 21136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:11.398588 21136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:11.399636 21136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce"
format_stamp: "Formatted at 2026-09-24 07:22:11 on dist-test-slave-rc1w"
server_key: "6991424e2999081b3f8067f66792df69"
server_key_iv: "418531ebb911a39e0c6399bcc5a5f8f4"
server_key_version: "encryptionkey@0"
I20260924 07:22:11.399991 21136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce"
format_stamp: "Formatted at 2026-09-24 07:22:11 on dist-test-slave-rc1w"
server_key: "6991424e2999081b3f8067f66792df69"
server_key_iv: "418531ebb911a39e0c6399bcc5a5f8f4"
server_key_version: "encryptionkey@0"
I20260924 07:22:11.401268 21136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:22:11.402019 21151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:11.402184 21136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:11.402277 21136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce"
format_stamp: "Formatted at 2026-09-24 07:22:11 on dist-test-slave-rc1w"
server_key: "6991424e2999081b3f8067f66792df69"
server_key_iv: "418531ebb911a39e0c6399bcc5a5f8f4"
server_key_version: "encryptionkey@0"
I20260924 07:22:11.402354 21136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:11.421396 21136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:11.425834 21136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:11.425984 21136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:11.426236 21136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:11.426610 21136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:11.426661 21136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:11.426692 21136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:11.426713 21136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:11.434303 21136 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:44467
I20260924 07:22:11.434341 21264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:44467 every 8 connection(s)
I20260924 07:22:11.434664 21136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:22:11.442884 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 21136
I20260924 07:22:11.442994 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:22:11.443238 4057 external_mini_cluster.cc:1428] Setting key 43bb686403b3223115aa4ddc4db8f543
Sep 24 07:22:11 dist-test-slave-rc1w krb5kdc[21029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234531, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:11.450068 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.435732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:35989 (local address 127.3.246.126:35403)
0924 07:22:11.435816 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:11.435818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:11.436079 (+ 261us) server_negotiation.cc:407] Connection header received
0924 07:22:11.436611 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:11.436613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:11.436626 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:11.436684 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:11.438545 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.439045 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.440864 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.441233 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.441312 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.444957 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:11.444958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:11.444959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:11.444967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:11.448308 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:11.448870 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:11.448871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:11.448871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:11.448936 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:11.449255 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:11.449256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:11.449256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:11.449457 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:11.449518 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:11.449885 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 07:22:11.449978 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:22:11.450274 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.435739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35403 (local address 127.3.246.65:35989)
0924 07:22:11.435901 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:11.435973 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:11.436492 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:11.436817 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:11.437035 (+ 218us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:11.437885 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:11.438177 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.438185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.439160 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.439161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.440745 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.440762 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.441401 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.441402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.441575 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.443030 (+ 1455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:11.443040 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:11.444848 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:11.448393 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:11.448401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:11.448403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:11.448777 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:11.449008 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:11.449009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:11.449009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:11.449166 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:11.449531 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:11.449538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:11.449720 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 07:22:11.449850 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 07:22:11.450015 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260924 07:22:11.450887 21265 heartbeater.cc:347] Connected to a master server at 127.3.246.126:35403
I20260924 07:22:11.451001 21265 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:11.451200 21265 heartbeater.cc:511] Master 127.3.246.126:35403 requested a full tablet report, sending...
I20260924 07:22:11.451805 21074 ts_manager.cc:194] Registered new tserver with Master: 1af9eb6b41c54ab1bb2c8be2714fa8ce (127.3.246.65:44467)
I20260924 07:22:11.452776 21074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:35989
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:22:11.476763 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:35403
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:11.559471 21270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:11.559672 21270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:11.559695 21270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:11.560743 21270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:11.560776 21270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:11.560801 21270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:22:11.562206 21270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:35403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:11.562376 21270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:11.562537 21270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:11.567396 21276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:11.567431 21275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:11.567443 21270 server_base.cc:1034] running on GCE node
W20260924 07:22:11.567549 21278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:11.567770 21270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:11.568018 21270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:11.569157 21270 hybrid_clock.cc:648] HybridClock initialized: now 1790234531569132 us; error 43 us; skew 500 ppm
Sep 24 07:22:11 dist-test-slave-rc1w krb5kdc[21029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234531, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:11.572084 21270 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:22:11.572543 21270 webserver.cc:460] Webserver started at http://127.3.246.66:44377/ using document root <none> and password file <none>
I20260924 07:22:11.572765 21270 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:11.572832 21270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:11.572965 21270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:11.574057 21270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "29a3c9f0238341d0a3832ea381b04b09"
format_stamp: "Formatted at 2026-09-24 07:22:11 on dist-test-slave-rc1w"
server_key: "c58bc94f1871b686e13c8e60077cdae2"
server_key_iv: "ca78faf9001491bd527e078d65eccfea"
server_key_version: "encryptionkey@0"
I20260924 07:22:11.574414 21270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "29a3c9f0238341d0a3832ea381b04b09"
format_stamp: "Formatted at 2026-09-24 07:22:11 on dist-test-slave-rc1w"
server_key: "c58bc94f1871b686e13c8e60077cdae2"
server_key_iv: "ca78faf9001491bd527e078d65eccfea"
server_key_version: "encryptionkey@0"
I20260924 07:22:11.575723 21270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:22:11.576431 21285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:11.576599 21270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:11.576696 21270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "29a3c9f0238341d0a3832ea381b04b09"
format_stamp: "Formatted at 2026-09-24 07:22:11 on dist-test-slave-rc1w"
server_key: "c58bc94f1871b686e13c8e60077cdae2"
server_key_iv: "ca78faf9001491bd527e078d65eccfea"
server_key_version: "encryptionkey@0"
I20260924 07:22:11.576776 21270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:11.594812 21270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:11.598085 21270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:11.598253 21270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:11.598488 21270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:11.598791 21270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:11.598845 21270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:11.598879 21270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:11.598908 21270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:11.604545 21270 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:33025
I20260924 07:22:11.604590 21398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:33025 every 8 connection(s)
I20260924 07:22:11.604939 21270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:22:11 dist-test-slave-rc1w krb5kdc[21029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234531, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:11.612452 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 21270
I20260924 07:22:11.612596 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:22:11.612802 4057 external_mini_cluster.cc:1428] Setting key efa1e365325b9caccb16a44a2d56f0c8
I20260924 07:22:11.615995 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.605612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:55267 (local address 127.3.246.126:35403)
0924 07:22:11.605658 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:11.605660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:11.605907 (+ 247us) server_negotiation.cc:407] Connection header received
0924 07:22:11.606418 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:11.606419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:11.606431 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:11.606489 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:11.607396 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.607889 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.608866 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.609061 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.609134 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.611647 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:11.611648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:11.611649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:11.611655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:11.614342 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:11.614970 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:11.614972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:11.614972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:11.615047 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:11.615373 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:11.615374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:11.615374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:11.615481 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:11.615537 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:11.615872 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 07:22:11.615934 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:22:11.616235 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.605582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35403 (local address 127.3.246.66:55267)
0924 07:22:11.605769 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:11.605832 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:11.606319 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:11.606560 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:11.606564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:11.607111 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:11.607293 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.607298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.607984 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.607985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.608775 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.608777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.609154 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.609155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.609272 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.610242 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:11.610250 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:11.611562 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:11.614427 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:11.614434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:11.614436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:11.614757 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:11.615160 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:11.615161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:11.615162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:11.615304 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:11.615553 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:11.615559 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:11.615733 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 07:22:11.615849 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 07:22:11.616028 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
I20260924 07:22:11.616667 21399 heartbeater.cc:347] Connected to a master server at 127.3.246.126:35403
I20260924 07:22:11.616753 21399 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:11.616942 21399 heartbeater.cc:511] Master 127.3.246.126:35403 requested a full tablet report, sending...
I20260924 07:22:11.617317 21074 ts_manager.cc:194] Registered new tserver with Master: 29a3c9f0238341d0a3832ea381b04b09 (127.3.246.66:33025)
I20260924 07:22:11.617797 21074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:55267
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:22:11.638417 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:35403
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:11.718196 21404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:11.718415 21404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:11.718453 21404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:11.719584 21404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:11.719648 21404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:11.719693 21404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:22:11.721115 21404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:35403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:11.721369 21404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:11.721582 21404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:11.726310 21409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:11.726342 21412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:11.726643 21404 server_base.cc:1034] running on GCE node
W20260924 07:22:11.726455 21410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:11.726883 21404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:11.727138 21404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:11.728325 21404 hybrid_clock.cc:648] HybridClock initialized: now 1790234531728310 us; error 46 us; skew 500 ppm
Sep 24 07:22:11 dist-test-slave-rc1w krb5kdc[21029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234531, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:11.731900 21404 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:22:11.732375 21404 webserver.cc:460] Webserver started at http://127.3.246.67:35263/ using document root <none> and password file <none>
I20260924 07:22:11.732610 21404 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:11.732654 21404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:11.732741 21404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:11.733799 21404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "f5c115075834426aabcef28d0a931c60"
format_stamp: "Formatted at 2026-09-24 07:22:11 on dist-test-slave-rc1w"
server_key: "796af8415d8e75725cd4199f28d7288c"
server_key_iv: "f6d5bee475715c3e61abe08b9b80c89f"
server_key_version: "encryptionkey@0"
I20260924 07:22:11.734117 21404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "f5c115075834426aabcef28d0a931c60"
format_stamp: "Formatted at 2026-09-24 07:22:11 on dist-test-slave-rc1w"
server_key: "796af8415d8e75725cd4199f28d7288c"
server_key_iv: "f6d5bee475715c3e61abe08b9b80c89f"
server_key_version: "encryptionkey@0"
I20260924 07:22:11.735355 21404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:22:11.735998 21419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:11.736178 21404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:22:11.736266 21404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "f5c115075834426aabcef28d0a931c60"
format_stamp: "Formatted at 2026-09-24 07:22:11 on dist-test-slave-rc1w"
server_key: "796af8415d8e75725cd4199f28d7288c"
server_key_iv: "f6d5bee475715c3e61abe08b9b80c89f"
server_key_version: "encryptionkey@0"
I20260924 07:22:11.736320 21404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:11.751842 21404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:11.755498 21404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:11.755642 21404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:11.755877 21404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:11.756179 21404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:11.756217 21404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:11.756268 21404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:11.756294 21404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:11.761930 21404 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:42709
I20260924 07:22:11.761997 21532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:42709 every 8 connection(s)
I20260924 07:22:11.762315 21404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:22:11.762912 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 21404
I20260924 07:22:11.763034 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:22:11.763233 4057 external_mini_cluster.cc:1428] Setting key 5340d26b77a45f5876fe33b502fd02a6
Sep 24 07:22:11 dist-test-slave-rc1w krb5kdc[21029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234531, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:11.773877 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.763025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:52121 (local address 127.3.246.126:35403)
0924 07:22:11.763113 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:11.763116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:11.763311 (+ 195us) server_negotiation.cc:407] Connection header received
0924 07:22:11.763992 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:11.763993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:11.764005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:11.764062 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:11.764984 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.765490 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.766557 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.766941 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.767061 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.770220 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:11.770221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:11.770222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:11.770229 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:11.772399 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:11.772846 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:11.772848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:11.772848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:11.772915 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:11.773182 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:11.773183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:11.773184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:11.773309 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:11.773384 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:11.773701 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 07:22:11.773786 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:22:11.774078 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.762993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:35403 (local address 127.3.246.67:52121)
0924 07:22:11.763181 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:11.763220 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:11.763884 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:11.764132 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:11.764137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:11.764699 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:11.764878 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.764882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.765591 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.765592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.766461 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.766464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.767073 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.767074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.767217 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.768669 (+ 1452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:11.768677 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:11.770092 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:11.772498 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:11.772506 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:11.772508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:11.772740 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:11.772985 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:11.772986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:11.772987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:11.773105 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:11.773375 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:11.773382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:11.773526 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 07:22:11.773643 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 07:22:11.773834 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
I20260924 07:22:11.774623 21533 heartbeater.cc:347] Connected to a master server at 127.3.246.126:35403
I20260924 07:22:11.774734 21533 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:11.774914 21533 heartbeater.cc:511] Master 127.3.246.126:35403 requested a full tablet report, sending...
I20260924 07:22:11.775302 21074 ts_manager.cc:194] Registered new tserver with Master: f5c115075834426aabcef28d0a931c60 (127.3.246.67:42709)
I20260924 07:22:11.775686 21074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:52121
I20260924 07:22:11.785753 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:22:11.793577 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.786997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.3.246.126:35403)
0924 07:22:11.787059 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:11.787061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:11.787120 (+ 59us) server_negotiation.cc:407] Connection header received
0924 07:22:11.787173 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:11.787174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:11.787183 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:11.787234 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:11.787719 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.788187 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.788849 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.789018 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.789064 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.790687 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:11.790688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:11.790688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:11.790694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:11.792571 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:11.792920 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:11.792921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:11.792921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:11.792968 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:11.793151 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:11.793151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:11.793152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:11.793288 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:11.793361 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:11.793442 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 07:22:11.793496 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:22:11.796375 21074 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:22:11.797964 21074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:22:11.804414 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.800174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:44467 (local address 127.0.0.1:53922)
0924 07:22:11.800410 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:11.800447 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:11.800517 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:11.800961 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:11.800962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:11.800964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:11.801002 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.801004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.802349 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.802350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.803266 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.803268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.804166 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.804167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.804252 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.804266 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 07:22:11.804330 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:22:11.804365 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260924 07:22:11.804476 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.800581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.3.246.65:44467)
0924 07:22:11.800795 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:11.800800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:11.800810 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:22:11.800849 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:11.800851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:11.800866 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:11.801075 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:11.801138 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.802256 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.803349 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.804083 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.804171 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.804255 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:11.804354 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 07:22:11.804393 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":117,"threads_started":1}
I20260924 07:22:11.805245 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.800201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:42709 (local address 127.0.0.1:44892)
0924 07:22:11.800488 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:11.800497 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:11.800536 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:11.800901 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:11.800904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:11.800907 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:11.800970 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.800972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.802285 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.802286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.803405 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.803407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.805001 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.805002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.805070 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.805074 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:11.805121 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 07:22:11.805171 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":25,"thread_start_us":62,"threads_started":1}
I20260924 07:22:11.805315 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.800175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:33025 (local address 127.0.0.1:43012)
0924 07:22:11.800453 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:11.800463 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:11.800521 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:11.801064 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:11.801065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:11.801067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:11.801112 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.801114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.802046 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.802047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.803405 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.803407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.805089 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.805089 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.805153 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.805156 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:22:11.805185 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 07:22:11.805218 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":37,"thread_start_us":44,"threads_started":1}
I20260924 07:22:11.805356 21199 tablet_service.cc:1467] Processing CreateTablet for tablet c82163f4c3b44dc4a28a825dc3ef8da6 (DEFAULT_TABLE table=test-table [id=17bbec7ad4ee43f8aef276fd9510c749]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:11.805531 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.800388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.3.246.67:42709)
0924 07:22:11.800656 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:11.800662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:11.800673 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:22:11.800710 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:11.800712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:11.800726 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:11.800808 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:11.801059 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.802200 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.804297 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.804878 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.804979 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.805030 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:11.805416 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 07:22:11.805457 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":77,"threads_started":1}
I20260924 07:22:11.805662 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.800270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.3.246.66:33025)
0924 07:22:11.800889 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:11.800894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:11.800903 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:22:11.800948 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:11.800949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:11.800964 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:11.801185 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:11.801230 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.801966 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.803488 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.804218 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.805442 (+ 1224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.805504 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:11.805527 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 07:22:11.805571 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":96,"threads_started":1}
I20260924 07:22:11.805750 21199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82163f4c3b44dc4a28a825dc3ef8da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:11.806131 21467 tablet_service.cc:1467] Processing CreateTablet for tablet c82163f4c3b44dc4a28a825dc3ef8da6 (DEFAULT_TABLE table=test-table [id=17bbec7ad4ee43f8aef276fd9510c749]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:11.806295 21333 tablet_service.cc:1467] Processing CreateTablet for tablet c82163f4c3b44dc4a28a825dc3ef8da6 (DEFAULT_TABLE table=test-table [id=17bbec7ad4ee43f8aef276fd9510c749]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:11.806465 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82163f4c3b44dc4a28a825dc3ef8da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:11.806622 21333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82163f4c3b44dc4a28a825dc3ef8da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:11.808971 21550 tablet_bootstrap.cc:492] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60: Bootstrap starting.
I20260924 07:22:11.809756 21550 tablet_bootstrap.cc:654] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:11.809813 21551 tablet_bootstrap.cc:492] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce: Bootstrap starting.
I20260924 07:22:11.810465 21550 log.cc:824] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:11.810513 21551 tablet_bootstrap.cc:654] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:11.811064 21551 log.cc:824] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:11.811260 21550 tablet_bootstrap.cc:492] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60: No bootstrap required, opened a new log
I20260924 07:22:11.811334 21550 ts_tablet_manager.cc:1397] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:22:11.811460 21552 tablet_bootstrap.cc:492] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09: Bootstrap starting.
I20260924 07:22:11.811830 21551 tablet_bootstrap.cc:492] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce: No bootstrap required, opened a new log
I20260924 07:22:11.811914 21551 ts_tablet_manager.cc:1397] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:22:11.812033 21552 tablet_bootstrap.cc:654] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:11.812592 21552 log.cc:824] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:11.814733 21552 tablet_bootstrap.cc:492] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09: No bootstrap required, opened a new log
I20260924 07:22:11.814811 21552 ts_tablet_manager.cc:1397] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:22:11.815013 21550 raft_consensus.cc:348] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 44467 } } peers { permanent_uuid: "29a3c9f0238341d0a3832ea381b04b09" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33025 } } peers { permanent_uuid: "f5c115075834426aabcef28d0a931c60" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42709 } }
I20260924 07:22:11.815201 21550 raft_consensus.cc:374] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:11.815237 21550 raft_consensus.cc:729] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5c115075834426aabcef28d0a931c60, State: Initialized, Role: FOLLOWER
I20260924 07:22:11.815378 21550 consensus_queue.cc:260] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 44467 } } peers { permanent_uuid: "29a3c9f0238341d0a3832ea381b04b09" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33025 } } peers { permanent_uuid: "f5c115075834426aabcef28d0a931c60" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42709 } }
I20260924 07:22:11.815668 21550 ts_tablet_manager.cc:1428] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260924 07:22:11.815800 21533 heartbeater.cc:503] Master 127.3.246.126:35403 was elected leader, sending a full tablet report...
I20260924 07:22:11.815654 21551 raft_consensus.cc:348] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 44467 } } peers { permanent_uuid: "29a3c9f0238341d0a3832ea381b04b09" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33025 } } peers { permanent_uuid: "f5c115075834426aabcef28d0a931c60" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42709 } }
I20260924 07:22:11.815853 21551 raft_consensus.cc:374] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:11.815900 21551 raft_consensus.cc:729] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af9eb6b41c54ab1bb2c8be2714fa8ce, State: Initialized, Role: FOLLOWER
I20260924 07:22:11.816008 21551 consensus_queue.cc:260] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 44467 } } peers { permanent_uuid: "29a3c9f0238341d0a3832ea381b04b09" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33025 } } peers { permanent_uuid: "f5c115075834426aabcef28d0a931c60" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42709 } }
I20260924 07:22:11.816313 21551 ts_tablet_manager.cc:1428] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 07:22:11.816337 21265 heartbeater.cc:503] Master 127.3.246.126:35403 was elected leader, sending a full tablet report...
I20260924 07:22:11.817126 21552 raft_consensus.cc:348] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 44467 } } peers { permanent_uuid: "29a3c9f0238341d0a3832ea381b04b09" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33025 } } peers { permanent_uuid: "f5c115075834426aabcef28d0a931c60" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42709 } }
I20260924 07:22:11.817314 21552 raft_consensus.cc:374] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:11.817370 21552 raft_consensus.cc:729] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a3c9f0238341d0a3832ea381b04b09, State: Initialized, Role: FOLLOWER
I20260924 07:22:11.817503 21552 consensus_queue.cc:260] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 44467 } } peers { permanent_uuid: "29a3c9f0238341d0a3832ea381b04b09" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33025 } } peers { permanent_uuid: "f5c115075834426aabcef28d0a931c60" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42709 } }
I20260924 07:22:11.817799 21552 ts_tablet_manager.cc:1428] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 07:22:11.817996 21399 heartbeater.cc:503] Master 127.3.246.126:35403 was elected leader, sending a full tablet report...
W20260924 07:22:11.855463 21400 tablet.cc:2369] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:22:11.932966 21558 raft_consensus.cc:482] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:22:11.933085 21558 raft_consensus.cc:504] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 44467 } } peers { permanent_uuid: "29a3c9f0238341d0a3832ea381b04b09" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33025 } } peers { permanent_uuid: "f5c115075834426aabcef28d0a931c60" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42709 } }
I20260924 07:22:11.933390 21558 leader_election.cc:288] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1af9eb6b41c54ab1bb2c8be2714fa8ce (127.3.246.65:44467), f5c115075834426aabcef28d0a931c60 (127.3.246.67:42709)
W20260924 07:22:11.936189 21266 tablet.cc:2369] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:22:11.936965 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.933894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:49369 (local address 127.3.246.65:44467)
0924 07:22:11.933937 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:11.933939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:11.934196 (+ 257us) server_negotiation.cc:407] Connection header received
0924 07:22:11.934228 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:11.934229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:11.934240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:11.934290 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:11.934585 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.935237 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.936160 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.936613 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.936693 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.936702 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:11.936847 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 07:22:11.936886 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:22:11.936967 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.933808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:35933 (local address 127.3.246.67:42709)
0924 07:22:11.933874 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:11.933876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:11.933886 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:22:11.933919 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:11.933920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:11.933930 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:11.934029 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:11.934192 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.934973 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.936070 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.936601 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.936692 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.936701 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:11.936862 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 07:22:11.936900 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:22:11.936983 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.933654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:42709 (local address 127.3.246.66:35933)
0924 07:22:11.933745 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:11.933757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:11.933804 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:11.934037 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:11.934039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:11.934041 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:11.934113 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.934116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.935103 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.935105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.935967 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.935970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.936689 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.936690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.936774 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.936777 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:22:11.936852 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:22:11.936899 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":5}
I20260924 07:22:11.937019 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.933654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:44467 (local address 127.3.246.66:49369)
0924 07:22:11.933954 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:11.933964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:11.934050 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:11.934335 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:11.934336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:11.934338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:11.934488 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.934491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.935300 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.935301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.936055 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:11.936057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.936705 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.936707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:11.936773 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.936777 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:11.936841 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:22:11.936886 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":54,"thread_start_us":114,"threads_started":1}
I20260924 07:22:11.937310 21487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82163f4c3b44dc4a28a825dc3ef8da6" candidate_uuid: "29a3c9f0238341d0a3832ea381b04b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5c115075834426aabcef28d0a931c60" is_pre_election: true
I20260924 07:22:11.937475 21487 raft_consensus.cc:2454] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29a3c9f0238341d0a3832ea381b04b09 in term 0.
I20260924 07:22:11.937628 21219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82163f4c3b44dc4a28a825dc3ef8da6" candidate_uuid: "29a3c9f0238341d0a3832ea381b04b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce" is_pre_election: true
I20260924 07:22:11.937748 21288 leader_election.cc:302] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a3c9f0238341d0a3832ea381b04b09, f5c115075834426aabcef28d0a931c60; no voters:
I20260924 07:22:11.937822 21219 raft_consensus.cc:2454] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29a3c9f0238341d0a3832ea381b04b09 in term 0.
I20260924 07:22:11.937879 21558 raft_consensus.cc:2781] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:22:11.937932 21558 raft_consensus.cc:482] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:22:11.937958 21558 raft_consensus.cc:3037] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:11.938778 21558 raft_consensus.cc:504] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 44467 } } peers { permanent_uuid: "29a3c9f0238341d0a3832ea381b04b09" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33025 } } peers { permanent_uuid: "f5c115075834426aabcef28d0a931c60" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42709 } }
I20260924 07:22:11.938915 21558 leader_election.cc:288] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [CANDIDATE]: Term 1 election: Requested vote from peers 1af9eb6b41c54ab1bb2c8be2714fa8ce (127.3.246.65:44467), f5c115075834426aabcef28d0a931c60 (127.3.246.67:42709)
I20260924 07:22:11.939090 21219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82163f4c3b44dc4a28a825dc3ef8da6" candidate_uuid: "29a3c9f0238341d0a3832ea381b04b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce"
I20260924 07:22:11.939091 21487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82163f4c3b44dc4a28a825dc3ef8da6" candidate_uuid: "29a3c9f0238341d0a3832ea381b04b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5c115075834426aabcef28d0a931c60"
I20260924 07:22:11.939186 21219 raft_consensus.cc:3037] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:11.939182 21487 raft_consensus.cc:3037] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:11.939971 21487 raft_consensus.cc:2454] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29a3c9f0238341d0a3832ea381b04b09 in term 1.
I20260924 07:22:11.940062 21219 raft_consensus.cc:2454] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29a3c9f0238341d0a3832ea381b04b09 in term 1.
I20260924 07:22:11.940145 21288 leader_election.cc:302] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a3c9f0238341d0a3832ea381b04b09, f5c115075834426aabcef28d0a931c60; no voters:
I20260924 07:22:11.940212 21558 raft_consensus.cc:2781] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:11.940357 21558 raft_consensus.cc:686] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [term 1 LEADER]: Becoming Leader. State: Replica: 29a3c9f0238341d0a3832ea381b04b09, State: Running, Role: LEADER
I20260924 07:22:11.940598 21558 consensus_queue.cc:237] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 44467 } } peers { permanent_uuid: "29a3c9f0238341d0a3832ea381b04b09" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33025 } } peers { permanent_uuid: "f5c115075834426aabcef28d0a931c60" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42709 } }
I20260924 07:22:11.941447 21074 catalog_manager.cc:5495] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29a3c9f0238341d0a3832ea381b04b09 (127.3.246.66). New cstate: current_term: 1 leader_uuid: "29a3c9f0238341d0a3832ea381b04b09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 44467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29a3c9f0238341d0a3832ea381b04b09" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5c115075834426aabcef28d0a931c60" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42709 } health_report { overall_health: UNKNOWN } } }
I20260924 07:22:11.948181 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.945904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.3.246.66:33025)
0924 07:22:11.945980 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:11.945982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:11.945990 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:22:11.946064 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:11.946065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:11.946081 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:11.946134 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:11.946246 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.946684 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.947448 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:11.947629 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:11.947685 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:11.947868 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:11.947952 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:11.948015 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:11.948068 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 07:22:11.948083 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:22:11.949955 21487 raft_consensus.cc:1261] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60 [term 1 FOLLOWER]: Refusing update from remote peer 29a3c9f0238341d0a3832ea381b04b09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:11.949955 21219 raft_consensus.cc:1261] T c82163f4c3b44dc4a28a825dc3ef8da6 P 1af9eb6b41c54ab1bb2c8be2714fa8ce [term 1 FOLLOWER]: Refusing update from remote peer 29a3c9f0238341d0a3832ea381b04b09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:11.950318 21560 consensus_queue.cc:1035] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af9eb6b41c54ab1bb2c8be2714fa8ce" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 44467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:11.950558 21558 consensus_queue.cc:1035] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5c115075834426aabcef28d0a931c60" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 42709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:11.952364 21563 mvcc.cc:205] Tried to move back new op lower bound from 7332800642864074752 to 7332800642829275136. Current Snapshot: MvccSnapshot[applied={T|T < 7332800642864074752}]
I20260924 07:22:11.952845 21565 mvcc.cc:205] Tried to move back new op lower bound from 7332800642864074752 to 7332800642829275136. Current Snapshot: MvccSnapshot[applied={T|T < 7332800642864074752}]
I20260924 07:22:11.953572 21564 mvcc.cc:205] Tried to move back new op lower bound from 7332800642864074752 to 7332800642829275136. Current Snapshot: MvccSnapshot[applied={T|T < 7332800642864074752 or (T in {7332800642864074752})}]
I20260924 07:22:11.955509 21073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:22:11.955596 21073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:22:11.956327 21073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45f906aed782472faf3edd89d9315b05: Sending DeleteTablet for 3 replicas of tablet c82163f4c3b44dc4a28a825dc3ef8da6
I20260924 07:22:11.956660 21199 tablet_service.cc:1514] Processing DeleteTablet for tablet c82163f4c3b44dc4a28a825dc3ef8da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:11 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:53922
I20260924 07:22:11.956660 21333 tablet_service.cc:1514] Processing DeleteTablet for tablet c82163f4c3b44dc4a28a825dc3ef8da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:11 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:43012
I20260924 07:22:11.956982 21467 tablet_service.cc:1514] Processing DeleteTablet for tablet c82163f4c3b44dc4a28a825dc3ef8da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:11 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:44892
I20260924 07:22:11.957188 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 21136
I20260924 07:22:11.957324 21575 tablet_replica.cc:330] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60: stopping tablet replica
I20260924 07:22:11.957420 21575 raft_consensus.cc:2229] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:22:11.957888 21575 raft_consensus.cc:2258] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:11.958635 21575 ts_tablet_manager.cc:1905] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:22:11.959759 21574 tablet_replica.cc:330] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09: stopping tablet replica
I20260924 07:22:11.959856 21574 raft_consensus.cc:2229] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:22:11.959969 21574 raft_consensus.cc:2258] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:11.960445 21574 ts_tablet_manager.cc:1905] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:22:11.960464 21575 ts_tablet_manager.cc:1918] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:22:11.960525 21575 log.cc:1196] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/c82163f4c3b44dc4a28a825dc3ef8da6
I20260924 07:22:11.960819 21575 ts_tablet_manager.cc:1939] T c82163f4c3b44dc4a28a825dc3ef8da6 P f5c115075834426aabcef28d0a931c60: Deleting consensus metadata
I20260924 07:22:11.961225 21060 catalog_manager.cc:4841] TS f5c115075834426aabcef28d0a931c60 (127.3.246.67:42709): tablet c82163f4c3b44dc4a28a825dc3ef8da6 (table test-table [id=17bbec7ad4ee43f8aef276fd9510c749]) successfully deleted
I20260924 07:22:11.962167 21574 ts_tablet_manager.cc:1918] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:22:11.962239 21574 log.cc:1196] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/c82163f4c3b44dc4a28a825dc3ef8da6
I20260924 07:22:11.962512 21574 ts_tablet_manager.cc:1939] T c82163f4c3b44dc4a28a825dc3ef8da6 P 29a3c9f0238341d0a3832ea381b04b09: Deleting consensus metadata
I20260924 07:22:11.962898 21059 catalog_manager.cc:4841] TS 29a3c9f0238341d0a3832ea381b04b09 (127.3.246.66:33025): tablet c82163f4c3b44dc4a28a825dc3ef8da6 (table test-table [id=17bbec7ad4ee43f8aef276fd9510c749]) successfully deleted
W20260924 07:22:11.965148 21061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:44467 (error 108)
I20260924 07:22:11.965345 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 21270
I20260924 07:22:11.966223 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:11.966102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:44467 (local address 127.0.0.1:53926)
0924 07:22:11.966156 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:11.966195 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:22:11.966373 21061 catalog_manager.cc:4584] TS 1af9eb6b41c54ab1bb2c8be2714fa8ce (127.3.246.65:44467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c82163f4c3b44dc4a28a825dc3ef8da6: Network error: Client connection negotiation failed: client connection to 127.3.246.65:44467: connect: Connection refused (error 111)
I20260924 07:22:11.970866 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 21404
I20260924 07:22:11.975489 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 21042
2026-09-24T07:22:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2009 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T07:22:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:11Z Disabled control of system clock
I20260924 07:22:12.002434 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:44709
--webserver_interface=127.3.246.126
--webserver_port=0
--builtin_ntp_servers=127.3.246.84:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:44709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:22:12.080786 21580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:12.080984 21580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:12.081003 21580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:12.082085 21580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:12.082118 21580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:12.082129 21580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:12.082149 21580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:12.082160 21580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:12.083389 21580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:44709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:44709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:12.083570 21580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:12.083781 21580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:12.088274 21588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:12.088310 21585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:12.088304 21586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:12.088598 21580 server_base.cc:1034] running on GCE node
I20260924 07:22:12.088868 21580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:12.089184 21580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:12.090368 21580 hybrid_clock.cc:648] HybridClock initialized: now 1790234532090343 us; error 34 us; skew 500 ppm
I20260924 07:22:12.092068 21580 webserver.cc:460] Webserver started at http://127.3.246.126:40979/ using document root <none> and password file <none>
I20260924 07:22:12.092311 21580 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:12.092379 21580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:12.092515 21580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:12.093534 21580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "47f151a5d2374d1a8bd0e9763ff887a3"
format_stamp: "Formatted at 2026-09-24 07:22:12 on dist-test-slave-rc1w"
server_key: "36922e90604108fa7dc1b76b7d8b9346"
server_key_iv: "4ca204afa6534539d47922e4bf97f1b4"
server_key_version: "encryptionkey@0"
I20260924 07:22:12.093912 21580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "47f151a5d2374d1a8bd0e9763ff887a3"
format_stamp: "Formatted at 2026-09-24 07:22:12 on dist-test-slave-rc1w"
server_key: "36922e90604108fa7dc1b76b7d8b9346"
server_key_iv: "4ca204afa6534539d47922e4bf97f1b4"
server_key_version: "encryptionkey@0"
I20260924 07:22:12.095201 21580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:22:12.095932 21594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:12.096104 21580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:22:12.096202 21580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "47f151a5d2374d1a8bd0e9763ff887a3"
format_stamp: "Formatted at 2026-09-24 07:22:12 on dist-test-slave-rc1w"
server_key: "36922e90604108fa7dc1b76b7d8b9346"
server_key_iv: "4ca204afa6534539d47922e4bf97f1b4"
server_key_version: "encryptionkey@0"
I20260924 07:22:12.096287 21580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:12.105567 21580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:12.105865 21580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:12.106007 21580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:12.109503 21580 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:44709
I20260924 07:22:12.109548 21646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:44709 every 8 connection(s)
I20260924 07:22:12.109905 21580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:12.110833 21647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:12.112957 21647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3: Bootstrap starting.
I20260924 07:22:12.113514 21647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:12.113844 21647 log.cc:824] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:12.114502 21647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3: No bootstrap required, opened a new log
I20260924 07:22:12.116091 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 21580
I20260924 07:22:12.116209 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:12.116115 21647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f151a5d2374d1a8bd0e9763ff887a3" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44709 } }
I20260924 07:22:12.116324 21647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:12.116392 21647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f151a5d2374d1a8bd0e9763ff887a3, State: Initialized, Role: FOLLOWER
I20260924 07:22:12.116446 4057 external_mini_cluster.cc:1428] Setting key 1cb804ba4a6b22d057eb9d4157a1b96c
I20260924 07:22:12.116541 21647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f151a5d2374d1a8bd0e9763ff887a3" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44709 } }
I20260924 07:22:12.116622 21647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:12.116657 21647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:12.116704 21647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:12.117353 21647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f151a5d2374d1a8bd0e9763ff887a3" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44709 } }
I20260924 07:22:12.117478 21647 leader_election.cc:302] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47f151a5d2374d1a8bd0e9763ff887a3; no voters:
I20260924 07:22:12.117652 21647 leader_election.cc:288] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:12.117770 21652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:12.117924 21647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:12.118006 21652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [term 1 LEADER]: Becoming Leader. State: Replica: 47f151a5d2374d1a8bd0e9763ff887a3, State: Running, Role: LEADER
I20260924 07:22:12.118161 21652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f151a5d2374d1a8bd0e9763ff887a3" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44709 } }
I20260924 07:22:12.118554 21653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47f151a5d2374d1a8bd0e9763ff887a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f151a5d2374d1a8bd0e9763ff887a3" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44709 } } }
I20260924 07:22:12.118566 21654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47f151a5d2374d1a8bd0e9763ff887a3. Latest consensus state: current_term: 1 leader_uuid: "47f151a5d2374d1a8bd0e9763ff887a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f151a5d2374d1a8bd0e9763ff887a3" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 44709 } } }
I20260924 07:22:12.118680 21653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:12.118686 21654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:12.118862 21661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:12.119297 21661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:12.121043 21661 catalog_manager.cc:1347] Generated new cluster ID: 894863c9c15d4b098a47af963cb4b0c8
I20260924 07:22:12.121095 21661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:12.123291 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.116957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37472 (local address 127.3.246.126:44709)
0924 07:22:12.117155 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:12.117161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:12.117185 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:22:12.117276 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:12.117281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:12.117334 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:12.117440 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:12.117784 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.119635 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.120557 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.121149 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.121261 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.122630 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:12.122634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:12.122635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:12.122815 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:12.122855 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:12.122867 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:12.122926 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:12.123024 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 07:22:12.123097 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260924 07:22:12.126076 21661 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:12.126765 21661 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:12.137362 21661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3: Generated new TSK 0
I20260924 07:22:12.137523 21661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:12.141608 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:44709
--builtin_ntp_servers=127.3.246.84:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:22:12.223336 21671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:12.223562 21671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:12.223598 21671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:12.224737 21671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:12.224800 21671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:12.224845 21671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:22:12.226292 21671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:44709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:12.226538 21671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:12.226828 21671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:12.231786 21679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:12.231846 21676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:12.231793 21677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:12.231994 21671 server_base.cc:1034] running on GCE node
I20260924 07:22:12.232306 21671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:12.232573 21671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:12.233740 21671 hybrid_clock.cc:648] HybridClock initialized: now 1790234532233720 us; error 35 us; skew 500 ppm
I20260924 07:22:12.235790 21671 webserver.cc:460] Webserver started at http://127.3.246.65:38433/ using document root <none> and password file <none>
I20260924 07:22:12.236040 21671 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:12.236117 21671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:12.236253 21671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:12.237284 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "17df63ae6c0348a1b0a43dcd87b74392"
format_stamp: "Formatted at 2026-09-24 07:22:12 on dist-test-slave-rc1w"
server_key: "7e8b2cf022d4d05d7ce0b41dff1a835b"
server_key_iv: "36812b31023e132d25b0340bca0aa804"
server_key_version: "encryptionkey@0"
I20260924 07:22:12.237643 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "17df63ae6c0348a1b0a43dcd87b74392"
format_stamp: "Formatted at 2026-09-24 07:22:12 on dist-test-slave-rc1w"
server_key: "7e8b2cf022d4d05d7ce0b41dff1a835b"
server_key_iv: "36812b31023e132d25b0340bca0aa804"
server_key_version: "encryptionkey@0"
I20260924 07:22:12.238987 21671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:22:12.239753 21685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:12.239905 21671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:12.239988 21671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "17df63ae6c0348a1b0a43dcd87b74392"
format_stamp: "Formatted at 2026-09-24 07:22:12 on dist-test-slave-rc1w"
server_key: "7e8b2cf022d4d05d7ce0b41dff1a835b"
server_key_iv: "36812b31023e132d25b0340bca0aa804"
server_key_version: "encryptionkey@0"
I20260924 07:22:12.240063 21671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:12.253187 21671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:12.253458 21671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:12.253618 21671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:12.253872 21671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:12.254199 21671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:12.254232 21671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:12.254252 21671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:12.254266 21671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:12.260043 21671 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:38187
I20260924 07:22:12.260092 21798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:38187 every 8 connection(s)
I20260924 07:22:12.260403 21671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:22:12.266139 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.261096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:55371 (local address 127.3.246.126:44709)
0924 07:22:12.261176 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:12.261178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:12.261410 (+ 232us) server_negotiation.cc:407] Connection header received
0924 07:22:12.261965 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:12.261966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:12.261978 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:12.262035 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:12.262696 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.263217 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.264197 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.264418 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.264504 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.265841 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:12.265842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:12.265842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:12.265847 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:12.265863 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:12.265867 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:12.265917 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:12.266020 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 07:22:12.266058 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:22:12.266342 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.261087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:44709 (local address 127.3.246.65:55371)
0924 07:22:12.261290 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:12.261322 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:12.261857 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:12.262110 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:12.262114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:12.262121 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:12.262598 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.262602 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.263324 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.263325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.264101 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.264102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.264515 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.264516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.264650 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.265616 (+ 966us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:22:12.265616 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:12.265645 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:22:12.265744 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:12.265924 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:12.265928 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:12.265932 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:12.266011 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:22:12.266122 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260924 07:22:12.266394 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 21671
I20260924 07:22:12.266490 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:22:12.266731 4057 external_mini_cluster.cc:1428] Setting key 54a106da08fefa7756ca9e37d530a971
I20260924 07:22:12.266912 21799 heartbeater.cc:347] Connected to a master server at 127.3.246.126:44709
I20260924 07:22:12.267000 21799 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:12.267208 21799 heartbeater.cc:511] Master 127.3.246.126:44709 requested a full tablet report, sending...
I20260924 07:22:12.267704 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:44709
--builtin_ntp_servers=127.3.246.84:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:22:12.267704 21611 ts_manager.cc:194] Registered new tserver with Master: 17df63ae6c0348a1b0a43dcd87b74392 (127.3.246.65:38187)
I20260924 07:22:12.268347 21611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.246.65:55371
W20260924 07:22:12.347213 21802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:12.347395 21802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:12.347416 21802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:12.348465 21802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:12.348495 21802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:12.348519 21802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:22:12.349833 21802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:44709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:12.349999 21802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:12.350229 21802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:12.354936 21810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:12.355054 21802 server_base.cc:1034] running on GCE node
W20260924 07:22:12.354961 21807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:12.354971 21808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:12.355353 21802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:12.355599 21802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:12.356740 21802 hybrid_clock.cc:648] HybridClock initialized: now 1790234532356726 us; error 42 us; skew 500 ppm
I20260924 07:22:12.358826 21802 webserver.cc:460] Webserver started at http://127.3.246.66:40725/ using document root <none> and password file <none>
I20260924 07:22:12.359102 21802 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:12.359150 21802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:12.359333 21802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:12.360344 21802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "002ad375b4aa4ef196c025f3ef709dee"
format_stamp: "Formatted at 2026-09-24 07:22:12 on dist-test-slave-rc1w"
server_key: "a0c3f6e649bfc627b199c4e3d34fe13b"
server_key_iv: "dff2e1e3a2b0ed59f98fa202e79ea1b5"
server_key_version: "encryptionkey@0"
I20260924 07:22:12.360688 21802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "002ad375b4aa4ef196c025f3ef709dee"
format_stamp: "Formatted at 2026-09-24 07:22:12 on dist-test-slave-rc1w"
server_key: "a0c3f6e649bfc627b199c4e3d34fe13b"
server_key_iv: "dff2e1e3a2b0ed59f98fa202e79ea1b5"
server_key_version: "encryptionkey@0"
I20260924 07:22:12.361959 21802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:22:12.362633 21816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:12.362807 21802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:22:12.362872 21802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "002ad375b4aa4ef196c025f3ef709dee"
format_stamp: "Formatted at 2026-09-24 07:22:12 on dist-test-slave-rc1w"
server_key: "a0c3f6e649bfc627b199c4e3d34fe13b"
server_key_iv: "dff2e1e3a2b0ed59f98fa202e79ea1b5"
server_key_version: "encryptionkey@0"
I20260924 07:22:12.362960 21802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:12.387466 21802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:12.387801 21802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:12.387945 21802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:12.388211 21802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:12.388562 21802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:12.388617 21802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:12.388651 21802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:12.388677 21802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:12.394392 21802 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:35471
I20260924 07:22:12.394443 21929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:35471 every 8 connection(s)
I20260924 07:22:12.394758 21802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:22:12.400763 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.395569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:59539 (local address 127.3.246.126:44709)
0924 07:22:12.395635 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:12.395637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:12.395881 (+ 244us) server_negotiation.cc:407] Connection header received
0924 07:22:12.396447 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:12.396449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:12.396465 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:12.396529 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:12.397214 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.397783 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.398760 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.399050 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.399124 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.400490 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:12.400491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:12.400491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:12.400495 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:12.400509 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:12.400514 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:12.400553 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:12.400654 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 07:22:12.400683 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:22:12.400992 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.395544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:44709 (local address 127.3.246.66:59539)
0924 07:22:12.395761 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:12.395793 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:12.396333 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:12.396599 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:12.396603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:12.396613 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:12.397099 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.397103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.397874 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.397875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.398665 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.398667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.399132 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.399134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.399273 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.400244 (+ 971us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:22:12.400245 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:12.400273 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:22:12.400296 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:12.400564 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:12.400569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:12.400573 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:12.400664 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 07:22:12.400770 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260924 07:22:12.401510 21930 heartbeater.cc:347] Connected to a master server at 127.3.246.126:44709
I20260924 07:22:12.401602 21930 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:12.401820 21930 heartbeater.cc:511] Master 127.3.246.126:44709 requested a full tablet report, sending...
I20260924 07:22:12.402236 21611 ts_manager.cc:194] Registered new tserver with Master: 002ad375b4aa4ef196c025f3ef709dee (127.3.246.66:35471)
I20260924 07:22:12.402658 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 21802
I20260924 07:22:12.402693 21611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.246.66:59539
I20260924 07:22:12.402750 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:22:12.402963 4057 external_mini_cluster.cc:1428] Setting key 8ae9dccc6395ec0d9bb3eec9f965cb11
I20260924 07:22:12.403826 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:44709
--builtin_ntp_servers=127.3.246.84:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:22:12.482686 21933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:12.482888 21933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:12.482909 21933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:12.484059 21933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:12.484098 21933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:12.484124 21933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:22:12.485567 21933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:44709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:12.485831 21933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:12.486124 21933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:12.491205 21939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:12.491180 21938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:12.491218 21933 server_base.cc:1034] running on GCE node
W20260924 07:22:12.491201 21941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:12.491604 21933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:12.491886 21933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:12.493042 21933 hybrid_clock.cc:648] HybridClock initialized: now 1790234532493044 us; error 37 us; skew 500 ppm
I20260924 07:22:12.494971 21933 webserver.cc:460] Webserver started at http://127.3.246.67:38895/ using document root <none> and password file <none>
I20260924 07:22:12.495227 21933 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:12.495294 21933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:12.495428 21933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:12.496958 21933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "7e86815e344444928c9884909ce89000"
format_stamp: "Formatted at 2026-09-24 07:22:12 on dist-test-slave-rc1w"
server_key: "b4fc43b5ae404602bc1735d8a4394607"
server_key_iv: "0a3d0864b6f84592af361b0784e1432d"
server_key_version: "encryptionkey@0"
I20260924 07:22:12.497437 21933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "7e86815e344444928c9884909ce89000"
format_stamp: "Formatted at 2026-09-24 07:22:12 on dist-test-slave-rc1w"
server_key: "b4fc43b5ae404602bc1735d8a4394607"
server_key_iv: "0a3d0864b6f84592af361b0784e1432d"
server_key_version: "encryptionkey@0"
I20260924 07:22:12.499531 21933 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 07:22:12.500432 21947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:12.500634 21933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:22:12.500742 21933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "7e86815e344444928c9884909ce89000"
format_stamp: "Formatted at 2026-09-24 07:22:12 on dist-test-slave-rc1w"
server_key: "b4fc43b5ae404602bc1735d8a4394607"
server_key_iv: "0a3d0864b6f84592af361b0784e1432d"
server_key_version: "encryptionkey@0"
I20260924 07:22:12.500813 21933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:12.510396 21933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:12.510614 21933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:12.510710 21933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:12.510860 21933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:12.511166 21933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:12.511221 21933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:12.511261 21933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:12.511288 21933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:12.516866 21933 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:37751
I20260924 07:22:12.516901 22060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:37751 every 8 connection(s)
I20260924 07:22:12.517230 21933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:22:12.518044 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 21933
I20260924 07:22:12.518136 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:22:12.518383 4057 external_mini_cluster.cc:1428] Setting key 9ed6699f846a6c28963d1ff28e136c2d
I20260924 07:22:12.523277 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.517993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:48451 (local address 127.3.246.126:44709)
0924 07:22:12.518084 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:12.518086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:12.518391 (+ 305us) server_negotiation.cc:407] Connection header received
0924 07:22:12.519152 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:12.519153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:12.519190 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:12.519250 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:12.519926 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.520409 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.521336 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.521510 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.521559 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.523096 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:12.523097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:12.523098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:12.523101 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:12.523111 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:12.523115 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:12.523167 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:12.523201 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 07:22:12.523222 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:22:12.523525 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.517991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:44709 (local address 127.3.246.67:48451)
0924 07:22:12.518232 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:12.518280 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:12.519051 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:12.519318 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:12.519328 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:12.519336 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:12.519851 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.519855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.520489 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.520489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.521266 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.521268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.521583 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.521584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.521732 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.522981 (+ 1249us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:22:12.522982 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:12.523008 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:22:12.523029 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:12.523156 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:12.523158 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:12.523160 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:22:12.523204 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 07:22:12.523282 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
I20260924 07:22:12.523985 22061 heartbeater.cc:347] Connected to a master server at 127.3.246.126:44709
I20260924 07:22:12.524070 22061 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:12.524251 22061 heartbeater.cc:511] Master 127.3.246.126:44709 requested a full tablet report, sending...
I20260924 07:22:12.524605 21611 ts_manager.cc:194] Registered new tserver with Master: 7e86815e344444928c9884909ce89000 (127.3.246.67:37751)
I20260924 07:22:12.524926 21611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.246.67:48451
I20260924 07:22:12.530035 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:22:12.535099 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.531358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.3.246.126:44709)
0924 07:22:12.531399 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:12.531402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:12.531554 (+ 152us) server_negotiation.cc:407] Connection header received
0924 07:22:12.531601 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:12.531602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:12.531610 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:12.531647 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:12.531764 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.532211 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.532985 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.533150 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.533210 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.534812 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:12.534813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:12.534814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:12.534825 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:12.534842 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:12.534847 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:12.534896 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:12.534994 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 07:22:12.535015 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:22:12.538039 21611 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:22:12.539531 21611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:22:12.546000 22073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.541916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.3.246.65:38187)
0924 07:22:12.542112 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:12.542117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:12.542238 (+ 121us) server_negotiation.cc:407] Connection header received
0924 07:22:12.542398 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:12.542400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:12.542415 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:12.542481 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:12.542710 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.543770 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.544874 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.545562 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.545640 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.545759 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:12.545876 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 07:22:12.545916 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
I20260924 07:22:12.546010 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.541650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:38187 (local address 127.0.0.1:49392)
0924 07:22:12.542170 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:12.542181 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:12.542268 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:12.542576 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:12.542577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:12.542579 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:12.542623 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.542624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.543848 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.543849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.544787 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.544789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.545661 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.545662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.545762 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.545777 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 07:22:12.545859 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 07:22:12.545895 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":128,"threads_started":1}
I20260924 07:22:12.546289 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.541908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.3.246.67:37751)
0924 07:22:12.542112 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:12.542117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:12.542229 (+ 112us) server_negotiation.cc:407] Connection header received
0924 07:22:12.542304 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:12.542306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:12.542322 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:12.542403 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:12.542610 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.543532 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.545108 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.545591 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.546036 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.546134 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:12.546164 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 07:22:12.546204 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":108,"threads_started":1}
I20260924 07:22:12.546020 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.541664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:37751 (local address 127.0.0.1:48692)
0924 07:22:12.542174 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:12.542184 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:12.542233 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:12.542473 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:12.542474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:12.542476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:12.542521 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.542523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.543634 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.543635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.544992 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.544995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.545733 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.545734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.545784 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.545787 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:22:12.545839 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 07:22:12.545895 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":38,"thread_start_us":113,"threads_started":1}
I20260924 07:22:12.546446 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.541634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:35471 (local address 127.0.0.1:37148)
0924 07:22:12.541958 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:12.541997 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:12.542069 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:12.542350 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:12.542352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:12.542355 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:12.542416 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.542418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.543752 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.543753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.544997 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.544999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.546198 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.546199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.546265 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.546269 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:12.546321 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 07:22:12.546365 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":43,"thread_start_us":86,"threads_started":1}
I20260924 07:22:12.546741 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.541757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.3.246.66:35471)
0924 07:22:12.541928 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:12.541932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:12.542049 (+ 117us) server_negotiation.cc:407] Connection header received
0924 07:22:12.542177 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:12.542179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:12.542200 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:12.542273 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:12.542497 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.543670 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.545086 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.545787 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.545876 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.545952 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:12.546593 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 07:22:12.546636 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260924 07:22:12.546800 21733 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6963fcbf8b440dae7ba9176fe72006 (DEFAULT_TABLE table=test-table [id=26dc6d6c5c014546bd739ddc6ddc2082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:12.547255 21733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6963fcbf8b440dae7ba9176fe72006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:12.547612 21864 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6963fcbf8b440dae7ba9176fe72006 (DEFAULT_TABLE table=test-table [id=26dc6d6c5c014546bd739ddc6ddc2082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:12.548048 21864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6963fcbf8b440dae7ba9176fe72006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:12.550257 21995 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6963fcbf8b440dae7ba9176fe72006 (DEFAULT_TABLE table=test-table [id=26dc6d6c5c014546bd739ddc6ddc2082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:12.550571 21995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6963fcbf8b440dae7ba9176fe72006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:12.550963 22075 tablet_bootstrap.cc:492] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392: Bootstrap starting.
I20260924 07:22:12.551676 22075 tablet_bootstrap.cc:654] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:12.551720 22076 tablet_bootstrap.cc:492] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee: Bootstrap starting.
I20260924 07:22:12.552053 22075 log.cc:824] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:12.552320 22076 tablet_bootstrap.cc:654] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:12.552644 22076 log.cc:824] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:12.552922 22075 tablet_bootstrap.cc:492] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392: No bootstrap required, opened a new log
I20260924 07:22:12.552994 22075 ts_tablet_manager.cc:1397] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:22:12.553326 22076 tablet_bootstrap.cc:492] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee: No bootstrap required, opened a new log
I20260924 07:22:12.553406 22076 ts_tablet_manager.cc:1397] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:22:12.553912 22079 tablet_bootstrap.cc:492] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000: Bootstrap starting.
I20260924 07:22:12.554576 22079 tablet_bootstrap.cc:654] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:12.554872 22079 log.cc:824] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:12.555142 22075 raft_consensus.cc:348] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "002ad375b4aa4ef196c025f3ef709dee" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35471 } } peers { permanent_uuid: "17df63ae6c0348a1b0a43dcd87b74392" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 38187 } } peers { permanent_uuid: "7e86815e344444928c9884909ce89000" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 37751 } }
I20260924 07:22:12.555328 22075 raft_consensus.cc:374] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:12.555374 22075 raft_consensus.cc:729] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17df63ae6c0348a1b0a43dcd87b74392, State: Initialized, Role: FOLLOWER
I20260924 07:22:12.555485 22075 consensus_queue.cc:260] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "002ad375b4aa4ef196c025f3ef709dee" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35471 } } peers { permanent_uuid: "17df63ae6c0348a1b0a43dcd87b74392" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 38187 } } peers { permanent_uuid: "7e86815e344444928c9884909ce89000" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 37751 } }
I20260924 07:22:12.555604 22079 tablet_bootstrap.cc:492] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000: No bootstrap required, opened a new log
I20260924 07:22:12.555670 22079 ts_tablet_manager.cc:1397] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:22:12.555616 22076 raft_consensus.cc:348] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "002ad375b4aa4ef196c025f3ef709dee" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35471 } } peers { permanent_uuid: "17df63ae6c0348a1b0a43dcd87b74392" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 38187 } } peers { permanent_uuid: "7e86815e344444928c9884909ce89000" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 37751 } }
I20260924 07:22:12.555771 22075 ts_tablet_manager.cc:1428] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 07:22:12.555783 22076 raft_consensus.cc:374] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:12.555816 22076 raft_consensus.cc:729] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002ad375b4aa4ef196c025f3ef709dee, State: Initialized, Role: FOLLOWER
I20260924 07:22:12.555874 21799 heartbeater.cc:503] Master 127.3.246.126:44709 was elected leader, sending a full tablet report...
I20260924 07:22:12.555918 22076 consensus_queue.cc:260] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "002ad375b4aa4ef196c025f3ef709dee" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35471 } } peers { permanent_uuid: "17df63ae6c0348a1b0a43dcd87b74392" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 38187 } } peers { permanent_uuid: "7e86815e344444928c9884909ce89000" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 37751 } }
I20260924 07:22:12.556164 22076 ts_tablet_manager.cc:1428] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 07:22:12.556282 21930 heartbeater.cc:503] Master 127.3.246.126:44709 was elected leader, sending a full tablet report...
I20260924 07:22:12.557878 22079 raft_consensus.cc:348] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "002ad375b4aa4ef196c025f3ef709dee" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35471 } } peers { permanent_uuid: "17df63ae6c0348a1b0a43dcd87b74392" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 38187 } } peers { permanent_uuid: "7e86815e344444928c9884909ce89000" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 37751 } }
I20260924 07:22:12.558058 22079 raft_consensus.cc:374] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:12.558109 22079 raft_consensus.cc:729] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e86815e344444928c9884909ce89000, State: Initialized, Role: FOLLOWER
I20260924 07:22:12.558238 22079 consensus_queue.cc:260] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "002ad375b4aa4ef196c025f3ef709dee" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35471 } } peers { permanent_uuid: "17df63ae6c0348a1b0a43dcd87b74392" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 38187 } } peers { permanent_uuid: "7e86815e344444928c9884909ce89000" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 37751 } }
I20260924 07:22:12.558508 22079 ts_tablet_manager.cc:1428] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:22:12.558643 22061 heartbeater.cc:503] Master 127.3.246.126:44709 was elected leader, sending a full tablet report...
W20260924 07:22:12.645344 21931 tablet.cc:2369] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:22:12.761129 21800 tablet.cc:2369] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:22:12.767779 22062 tablet.cc:2369] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:22:12.807374 22083 raft_consensus.cc:482] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:22:12.807507 22083 raft_consensus.cc:504] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "002ad375b4aa4ef196c025f3ef709dee" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35471 } } peers { permanent_uuid: "17df63ae6c0348a1b0a43dcd87b74392" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 38187 } } peers { permanent_uuid: "7e86815e344444928c9884909ce89000" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 37751 } }
I20260924 07:22:12.807921 22083 leader_election.cc:288] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 002ad375b4aa4ef196c025f3ef709dee (127.3.246.66:35471), 17df63ae6c0348a1b0a43dcd87b74392 (127.3.246.65:38187)
I20260924 07:22:12.811987 22073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.808254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:38903 (local address 127.3.246.65:38187)
0924 07:22:12.808302 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:12.808305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:12.808316 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:22:12.808352 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:12.808353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:12.808388 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:12.808476 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:12.808731 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.809496 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.811032 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.811606 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.811688 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.811696 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:12.811863 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 07:22:12.811906 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:22:12.811981 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.808278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:60905 (local address 127.3.246.66:35471)
0924 07:22:12.808320 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:12.808322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:12.808398 (+ 76us) server_negotiation.cc:407] Connection header received
0924 07:22:12.808425 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:12.808426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:12.808434 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:12.808490 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:12.808732 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.809487 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.811033 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.811609 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.811688 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.811696 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:12.811863 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 07:22:12.811906 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:22:12.811997 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.808137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:38187 (local address 127.3.246.67:38903)
0924 07:22:12.808203 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:12.808217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:12.808265 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:12.808506 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:12.808507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:12.808508 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:12.808611 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.808613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.809575 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.809576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.810923 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.810926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.811700 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.811701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.811778 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.811781 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:22:12.811839 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 07:22:12.811892 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":95}
I20260924 07:22:12.812003 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.808137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:35471 (local address 127.3.246.67:60905)
0924 07:22:12.808316 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:12.808325 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:12.808381 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:12.808506 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:12.808507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:12.808508 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:12.808617 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.808618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.809575 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.809576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.810923 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:12.810926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.811701 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.811701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:12.811780 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.811782 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:22:12.811839 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 07:22:12.811892 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":79,"thread_start_us":96,"threads_started":1}
I20260924 07:22:12.812433 21884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6963fcbf8b440dae7ba9176fe72006" candidate_uuid: "7e86815e344444928c9884909ce89000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002ad375b4aa4ef196c025f3ef709dee" is_pre_election: true
I20260924 07:22:12.812428 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6963fcbf8b440dae7ba9176fe72006" candidate_uuid: "7e86815e344444928c9884909ce89000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17df63ae6c0348a1b0a43dcd87b74392" is_pre_election: true
I20260924 07:22:12.812589 21753 raft_consensus.cc:2454] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e86815e344444928c9884909ce89000 in term 0.
I20260924 07:22:12.812589 21884 raft_consensus.cc:2454] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e86815e344444928c9884909ce89000 in term 0.
I20260924 07:22:12.812819 21948 leader_election.cc:302] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 002ad375b4aa4ef196c025f3ef709dee, 7e86815e344444928c9884909ce89000; no voters:
I20260924 07:22:12.812963 22083 raft_consensus.cc:2781] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:22:12.813020 22083 raft_consensus.cc:482] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:22:12.813047 22083 raft_consensus.cc:3037] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:12.813860 22083 raft_consensus.cc:504] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "002ad375b4aa4ef196c025f3ef709dee" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35471 } } peers { permanent_uuid: "17df63ae6c0348a1b0a43dcd87b74392" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 38187 } } peers { permanent_uuid: "7e86815e344444928c9884909ce89000" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 37751 } }
I20260924 07:22:12.814005 22083 leader_election.cc:288] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [CANDIDATE]: Term 1 election: Requested vote from peers 002ad375b4aa4ef196c025f3ef709dee (127.3.246.66:35471), 17df63ae6c0348a1b0a43dcd87b74392 (127.3.246.65:38187)
I20260924 07:22:12.814182 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6963fcbf8b440dae7ba9176fe72006" candidate_uuid: "7e86815e344444928c9884909ce89000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17df63ae6c0348a1b0a43dcd87b74392"
I20260924 07:22:12.814219 21884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6963fcbf8b440dae7ba9176fe72006" candidate_uuid: "7e86815e344444928c9884909ce89000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002ad375b4aa4ef196c025f3ef709dee"
I20260924 07:22:12.814271 21753 raft_consensus.cc:3037] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:12.814282 21884 raft_consensus.cc:3037] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:12.815196 21753 raft_consensus.cc:2454] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e86815e344444928c9884909ce89000 in term 1.
I20260924 07:22:12.815196 21884 raft_consensus.cc:2454] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e86815e344444928c9884909ce89000 in term 1.
I20260924 07:22:12.815399 21949 leader_election.cc:302] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17df63ae6c0348a1b0a43dcd87b74392, 7e86815e344444928c9884909ce89000; no voters:
I20260924 07:22:12.815531 22083 raft_consensus.cc:2781] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:12.815723 22083 raft_consensus.cc:686] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [term 1 LEADER]: Becoming Leader. State: Replica: 7e86815e344444928c9884909ce89000, State: Running, Role: LEADER
I20260924 07:22:12.815811 22083 consensus_queue.cc:237] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "002ad375b4aa4ef196c025f3ef709dee" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35471 } } peers { permanent_uuid: "17df63ae6c0348a1b0a43dcd87b74392" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 38187 } } peers { permanent_uuid: "7e86815e344444928c9884909ce89000" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 37751 } }
I20260924 07:22:12.816769 21610 catalog_manager.cc:5495] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e86815e344444928c9884909ce89000 (127.3.246.67). New cstate: current_term: 1 leader_uuid: "7e86815e344444928c9884909ce89000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "002ad375b4aa4ef196c025f3ef709dee" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17df63ae6c0348a1b0a43dcd87b74392" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 38187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e86815e344444928c9884909ce89000" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 37751 } health_report { overall_health: HEALTHY } } }
I20260924 07:22:12.828985 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.826565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.3.246.67:37751)
0924 07:22:12.826607 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:12.826610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:12.826655 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:22:12.826745 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:12.826746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:12.826771 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:12.826847 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:12.826956 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.827453 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.828209 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:12.828406 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:12.828483 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:12.828652 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:12.828752 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:12.828815 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:12.828855 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 07:22:12.828873 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:22:12.830780 21884 raft_consensus.cc:1261] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee [term 1 FOLLOWER]: Refusing update from remote peer 7e86815e344444928c9884909ce89000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:12.830826 21753 raft_consensus.cc:1261] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392 [term 1 FOLLOWER]: Refusing update from remote peer 7e86815e344444928c9884909ce89000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:12.831203 22085 consensus_queue.cc:1035] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002ad375b4aa4ef196c025f3ef709dee" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:12.831324 22083 consensus_queue.cc:1035] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17df63ae6c0348a1b0a43dcd87b74392" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 38187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:12.834616 22090 mvcc.cc:205] Tried to move back new op lower bound from 7332800646471966720 to 7332800646414192640. Current Snapshot: MvccSnapshot[applied={T|T < 7332800646471966720}]
I20260924 07:22:12.835968 21611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:22:12.836066 21611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:22:12.836786 21611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47f151a5d2374d1a8bd0e9763ff887a3: Sending DeleteTablet for 3 replicas of tablet 1a6963fcbf8b440dae7ba9176fe72006
I20260924 07:22:12.837158 21733 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6963fcbf8b440dae7ba9176fe72006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:12 UTC) from {username='slave'} at 127.0.0.1:49392
I20260924 07:22:12.837189 21864 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6963fcbf8b440dae7ba9176fe72006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:12 UTC) from {username='slave'} at 127.0.0.1:37148
I20260924 07:22:12.837430 22099 tablet_replica.cc:330] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee: stopping tablet replica
I20260924 07:22:12.837412 21995 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6963fcbf8b440dae7ba9176fe72006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:12 UTC) from {username='slave'} at 127.0.0.1:48692
I20260924 07:22:12.837538 22099 raft_consensus.cc:2229] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:22:12.837641 22099 raft_consensus.cc:2258] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:12.837664 22100 tablet_replica.cc:330] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000: stopping tablet replica
I20260924 07:22:12.837844 22098 tablet_replica.cc:330] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392: stopping tablet replica
I20260924 07:22:12.837920 22100 raft_consensus.cc:2229] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:22:12.837934 22098 raft_consensus.cc:2229] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:22:12.837949 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 21671
I20260924 07:22:12.838048 22098 raft_consensus.cc:2258] T 1a6963fcbf8b440dae7ba9176fe72006 P 17df63ae6c0348a1b0a43dcd87b74392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:12.838975 22099 ts_tablet_manager.cc:1905] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:22:12.839550 22100 raft_consensus.cc:2258] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:12.839938 22100 ts_tablet_manager.cc:1905] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:22:12.840947 22099 ts_tablet_manager.cc:1918] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:22:12.841023 22099 log.cc:1196] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/1a6963fcbf8b440dae7ba9176fe72006
I20260924 07:22:12.841277 22099 ts_tablet_manager.cc:1939] T 1a6963fcbf8b440dae7ba9176fe72006 P 002ad375b4aa4ef196c025f3ef709dee: Deleting consensus metadata
I20260924 07:22:12.841295 22100 ts_tablet_manager.cc:1918] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:22:12.841372 22100 log.cc:1196] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/1a6963fcbf8b440dae7ba9176fe72006
I20260924 07:22:12.841642 22100 ts_tablet_manager.cc:1939] T 1a6963fcbf8b440dae7ba9176fe72006 P 7e86815e344444928c9884909ce89000: Deleting consensus metadata
I20260924 07:22:12.841717 21595 catalog_manager.cc:4841] TS 002ad375b4aa4ef196c025f3ef709dee (127.3.246.66:35471): tablet 1a6963fcbf8b440dae7ba9176fe72006 (table test-table [id=26dc6d6c5c014546bd739ddc6ddc2082]) successfully deleted
I20260924 07:22:12.842034 21597 catalog_manager.cc:4841] TS 7e86815e344444928c9884909ce89000 (127.3.246.67:37751): tablet 1a6963fcbf8b440dae7ba9176fe72006 (table test-table [id=26dc6d6c5c014546bd739ddc6ddc2082]) successfully deleted
W20260924 07:22:12.843907 21596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:38187 (error 108)
I20260924 07:22:12.844130 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 21802
I20260924 07:22:12.844707 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:12.844218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:38187 (local address 127.0.0.1:49398)
0924 07:22:12.844631 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:12.844670 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393}
W20260924 07:22:12.845559 21596 catalog_manager.cc:4584] TS 17df63ae6c0348a1b0a43dcd87b74392 (127.3.246.65:38187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a6963fcbf8b440dae7ba9176fe72006: Network error: Client connection negotiation failed: client connection to 127.3.246.65:38187: connect: Connection refused (error 111)
I20260924 07:22:12.848801 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 21933
I20260924 07:22:12.853432 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 21580
2026-09-24T07:22:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (877 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:12 dist-test-slave-rc1w krb5kdc[22104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:12 dist-test-slave-rc1w krb5kdc[22104](info): set up 2 sockets
Sep 24 07:22:12 dist-test-slave-rc1w krb5kdc[22104](info): commencing operation
krb5kdc: starting...
W20260924 07:22:13.897550 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:13 dist-test-slave-rc1w krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:22:14.010527 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:34707
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:34707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:14.089648 22117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:14.089926 22117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:14.089960 22117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:14.091133 22117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:14.091210 22117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:14.091234 22117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:14.091264 22117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:14.091295 22117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:14.092763 22117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:34707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:34707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:14.093035 22117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:14.093262 22117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:14.098280 22122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:14.098256 22125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:14.098326 22117 server_base.cc:1034] running on GCE node
W20260924 07:22:14.098289 22123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:14.098699 22117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:14.099007 22117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:14.100199 22117 hybrid_clock.cc:648] HybridClock initialized: now 1790234534100196 us; error 56 us; skew 500 ppm
Sep 24 07:22:14 dist-test-slave-rc1w krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:14.103598 22117 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:22:14.104056 22117 webserver.cc:460] Webserver started at http://127.3.246.126:41191/ using document root <none> and password file <none>
I20260924 07:22:14.104292 22117 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:14.104359 22117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:14.104532 22117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:14.105523 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "82a1b4285890438e8350e6fb6385d9c2"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "91b2ca185b09aeb1c056d31699279058"
server_key_iv: "ee23a63c81afa18fc1a012ee459db311"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.105891 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "82a1b4285890438e8350e6fb6385d9c2"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "91b2ca185b09aeb1c056d31699279058"
server_key_iv: "ee23a63c81afa18fc1a012ee459db311"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.107195 22117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:22:14.107913 22132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:14.108132 22117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:22:14.108222 22117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "82a1b4285890438e8350e6fb6385d9c2"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "91b2ca185b09aeb1c056d31699279058"
server_key_iv: "ee23a63c81afa18fc1a012ee459db311"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.108273 22117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:14.122992 22117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:14.126928 22117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:14.127044 22117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:14.130906 22117 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:34707
I20260924 07:22:14.130981 22184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:34707 every 8 connection(s)
I20260924 07:22:14.131330 22117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:14.132179 22185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:14.134542 22185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2: Bootstrap starting.
I20260924 07:22:14.134959 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 22117
I20260924 07:22:14.135100 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:14.135186 22185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:14.135313 4057 external_mini_cluster.cc:1428] Setting key bb98e0327123849bea7cf93cb30dba72
I20260924 07:22:14.135542 22185 log.cc:824] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:14.136379 22185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2: No bootstrap required, opened a new log
I20260924 07:22:14.137902 22185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a1b4285890438e8350e6fb6385d9c2" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34707 } }
I20260924 07:22:14.138055 22185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:14.138118 22185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a1b4285890438e8350e6fb6385d9c2, State: Initialized, Role: FOLLOWER
I20260924 07:22:14.138260 22185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a1b4285890438e8350e6fb6385d9c2" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34707 } }
I20260924 07:22:14.138339 22185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:14.138381 22185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:14.138428 22185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:14.138952 22185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a1b4285890438e8350e6fb6385d9c2" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34707 } }
I20260924 07:22:14.139070 22185 leader_election.cc:302] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82a1b4285890438e8350e6fb6385d9c2; no voters:
I20260924 07:22:14.139256 22185 leader_election.cc:288] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:14.139308 22190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:14.139449 22190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [term 1 LEADER]: Becoming Leader. State: Replica: 82a1b4285890438e8350e6fb6385d9c2, State: Running, Role: LEADER
I20260924 07:22:14.139520 22185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:14.139534 22190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a1b4285890438e8350e6fb6385d9c2" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34707 } }
I20260924 07:22:14.140017 22192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82a1b4285890438e8350e6fb6385d9c2. Latest consensus state: current_term: 1 leader_uuid: "82a1b4285890438e8350e6fb6385d9c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a1b4285890438e8350e6fb6385d9c2" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34707 } } }
I20260924 07:22:14.140187 22192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:14.140647 22199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:14.141135 22199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:14.141873 22191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82a1b4285890438e8350e6fb6385d9c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a1b4285890438e8350e6fb6385d9c2" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34707 } } }
I20260924 07:22:14.141980 22191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:14.142825 22199 catalog_manager.cc:1347] Generated new cluster ID: a485f728daae46f0bef3b5ef143703e2
I20260924 07:22:14.142874 22199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:22:14 dist-test-slave-rc1w krb5kdc[22104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:14.150002 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.135991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.3.246.126:34707)
0924 07:22:14.136126 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:14.136135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:14.136173 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:22:14.136677 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:14.136684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:14.136728 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:14.136869 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:14.137516 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.138933 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.140032 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.140647 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.140765 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.144627 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:14.144630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:14.144631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:14.144642 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:14.147865 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:14.148267 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:14.148271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:14.148271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:14.148370 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:14.148576 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:14.148576 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:14.148577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:14.148891 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:14.149036 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:14.149551 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 07:22:14.149731 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
I20260924 07:22:14.155840 22199 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:14.156533 22199 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:14.165076 22199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2: Generated new TSK 0
I20260924 07:22:14.165278 22199 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:22:14.192816 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34707
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:14.275300 22211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:14.275503 22211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:14.275530 22211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:14.276736 22211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:14.276782 22211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:14.276810 22211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:22:14.278510 22211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:14.278779 22211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:14.279043 22211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:14.284051 22217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:14.284179 22219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:14.284103 22216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:14.284200 22211 server_base.cc:1034] running on GCE node
I20260924 07:22:14.284504 22211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:14.284715 22211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:14.285876 22211 hybrid_clock.cc:648] HybridClock initialized: now 1790234534285866 us; error 39 us; skew 500 ppm
Sep 24 07:22:14 dist-test-slave-rc1w krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:14.288715 22211 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:22:14.289078 22211 webserver.cc:460] Webserver started at http://127.3.246.65:38199/ using document root <none> and password file <none>
I20260924 07:22:14.289270 22211 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:14.289310 22211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:14.289397 22211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:14.290450 22211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "43bc3609370a4b7e8a5bc5b0ff6df227"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "66f2cf1e8d79329217da8bb0fd5f2167"
server_key_iv: "85974bba0e77117571156e434170e5ee"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.290750 22211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "43bc3609370a4b7e8a5bc5b0ff6df227"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "66f2cf1e8d79329217da8bb0fd5f2167"
server_key_iv: "85974bba0e77117571156e434170e5ee"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.291986 22211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:22:14.292718 22226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:14.292932 22211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:22:14.293000 22211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "43bc3609370a4b7e8a5bc5b0ff6df227"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "66f2cf1e8d79329217da8bb0fd5f2167"
server_key_iv: "85974bba0e77117571156e434170e5ee"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.293092 22211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:14.303133 22211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:14.306917 22211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:14.307065 22211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:14.307333 22211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:14.307672 22211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:14.307727 22211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:14.307761 22211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:14.307788 22211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:14.313861 22211 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:43803
I20260924 07:22:14.313931 22339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:43803 every 8 connection(s)
I20260924 07:22:14.314273 22211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:22:14.317754 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 22211
I20260924 07:22:14.317867 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:22:14.318120 4057 external_mini_cluster.cc:1428] Setting key 4cd8e534a75318b83df0a19ad7750b4d
Sep 24 07:22:14 dist-test-slave-rc1w krb5kdc[22104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:14.326735 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.315095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:34411 (local address 127.3.246.126:34707)
0924 07:22:14.315184 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:14.315187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:14.315361 (+ 174us) server_negotiation.cc:407] Connection header received
0924 07:22:14.315844 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:14.315845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:14.315855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:14.315909 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:14.316788 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.317285 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.318722 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.318951 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.319003 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.322082 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:14.322084 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:14.322085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:14.322092 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:14.325195 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:14.325619 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:14.325621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:14.325621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:14.325774 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:14.326030 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:14.326031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:14.326031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:14.326187 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:14.326266 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:14.326513 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 07:22:14.326610 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:22:14.326759 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.315063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:34707 (local address 127.3.246.65:34411)
0924 07:22:14.315239 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:14.315274 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:14.315759 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:14.315960 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:14.315963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:14.316526 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:14.316714 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.316718 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.317362 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.317362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.318630 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.318633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.319032 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.319033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.319186 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.320252 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:14.320263 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:14.321985 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:14.325291 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:14.325299 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:14.325301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:14.325537 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:14.325856 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:14.325857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:14.325857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:14.325961 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:14.326258 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:14.326263 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:14.326378 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 07:22:14.326462 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:22:14.326589 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
I20260924 07:22:14.327419 22340 heartbeater.cc:347] Connected to a master server at 127.3.246.126:34707
I20260924 07:22:14.327546 22340 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:14.327750 22340 heartbeater.cc:511] Master 127.3.246.126:34707 requested a full tablet report, sending...
I20260924 07:22:14.328291 22149 ts_manager.cc:194] Registered new tserver with Master: 43bc3609370a4b7e8a5bc5b0ff6df227 (127.3.246.65:43803)
I20260924 07:22:14.328991 22149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:34411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:22:14.345304 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34707
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:14.426720 22345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:14.426991 22345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:14.427037 22345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:14.428208 22345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:14.428288 22345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:14.428333 22345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:22:14.429821 22345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:14.430120 22345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:14.430370 22345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:14.435437 22350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:14.435554 22345 server_base.cc:1034] running on GCE node
W20260924 07:22:14.435504 22353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:14.435648 22351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:14.435878 22345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:14.436192 22345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:14.437397 22345 hybrid_clock.cc:648] HybridClock initialized: now 1790234534437385 us; error 32 us; skew 500 ppm
Sep 24 07:22:14 dist-test-slave-rc1w krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:14.440488 22345 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:22:14.440925 22345 webserver.cc:460] Webserver started at http://127.3.246.66:43647/ using document root <none> and password file <none>
I20260924 07:22:14.441155 22345 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:14.441226 22345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:14.441380 22345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:14.442476 22345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "639c1854a1684fb782f717226d719747"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "2cc3d81fd7858e342766e511cba1c468"
server_key_iv: "5bd406bc0582dcc933b3142330947174"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.442829 22345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "639c1854a1684fb782f717226d719747"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "2cc3d81fd7858e342766e511cba1c468"
server_key_iv: "5bd406bc0582dcc933b3142330947174"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.444197 22345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:22:14.444967 22360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:14.445204 22345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:22:14.445297 22345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "639c1854a1684fb782f717226d719747"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "2cc3d81fd7858e342766e511cba1c468"
server_key_iv: "5bd406bc0582dcc933b3142330947174"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.445350 22345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:14.466872 22345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:14.471383 22345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:14.471501 22345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:14.471698 22345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:14.472039 22345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:14.472080 22345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:14.472126 22345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:14.472153 22345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:14.477830 22345 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:37113
I20260924 07:22:14.477912 22473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:37113 every 8 connection(s)
I20260924 07:22:14.478320 22345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:22:14.480244 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 22345
I20260924 07:22:14.480340 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:22:14.480566 4057 external_mini_cluster.cc:1428] Setting key 06e9f235fdafa41e0d4ccf3be18bee42
Sep 24 07:22:14 dist-test-slave-rc1w krb5kdc[22104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
I20260924 07:22:14.490911 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.479077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:33673 (local address 127.3.246.126:34707)
0924 07:22:14.479134 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:14.479138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:14.479351 (+ 213us) server_negotiation.cc:407] Connection header received
0924 07:22:14.479869 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:14.479870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:14.479881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:14.479937 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:14.481084 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.481659 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.483281 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.483608 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.483689 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.486926 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:14.486928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:14.486928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:14.486937 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:14.489642 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:14.490102 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:14.490103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:14.490103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:14.490155 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:14.490373 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:14.490373 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:14.490373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:14.490475 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:14.490526 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:14.490776 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 07:22:14.490841 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:22:14.491137 22476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.479061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:34707 (local address 127.3.246.66:33673)
0924 07:22:14.479223 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:14.479259 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:14.479756 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:14.480001 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:14.480004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:14.480748 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:14.480992 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.480999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.481810 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.481811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.483190 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.483194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.483698 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.483699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.483798 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.485044 (+ 1246us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:14.485057 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:14.486824 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:14.489759 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:14.489767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:14.489769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:14.490021 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:14.490220 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:14.490220 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:14.490221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:14.490319 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:14.490537 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:14.490543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:14.490671 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 07:22:14.490757 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 07:22:14.490879 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260924 07:22:14.491622 22474 heartbeater.cc:347] Connected to a master server at 127.3.246.126:34707
I20260924 07:22:14.491770 22474 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:14.491974 22474 heartbeater.cc:511] Master 127.3.246.126:34707 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
I20260924 07:22:14.492362 22149 ts_manager.cc:194] Registered new tserver with Master: 639c1854a1684fb782f717226d719747 (127.3.246.66:37113)
I20260924 07:22:14.492738 22149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:33673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:22:14.509573 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34707
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:14.589514 22479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:14.589787 22479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:14.589826 22479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:14.590977 22479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:14.591051 22479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:14.591096 22479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:22:14.592494 22479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:14.592757 22479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:14.592986 22479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:14.597944 22487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:14.598033 22484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:14.597944 22485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:14.598201 22479 server_base.cc:1034] running on GCE node
I20260924 07:22:14.598395 22479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:14.598688 22479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:14.599889 22479 hybrid_clock.cc:648] HybridClock initialized: now 1790234534599866 us; error 44 us; skew 500 ppm
Sep 24 07:22:14 dist-test-slave-rc1w krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:14.603051 22479 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:22:14.603500 22479 webserver.cc:460] Webserver started at http://127.3.246.67:36697/ using document root <none> and password file <none>
I20260924 07:22:14.603685 22479 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:14.603724 22479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:14.603853 22479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:14.604849 22479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "833ac764dedc4f3d9215ecd74fe4fcdf"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "cd12c564bb0e13379bb01ac9937a707f"
server_key_iv: "ecad1dc9f2cf2de504c3834fc7c5d9f6"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.605144 22479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "833ac764dedc4f3d9215ecd74fe4fcdf"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "cd12c564bb0e13379bb01ac9937a707f"
server_key_iv: "ecad1dc9f2cf2de504c3834fc7c5d9f6"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.606421 22479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:22:14.607049 22494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:14.607230 22479 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:22:14.607314 22479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "833ac764dedc4f3d9215ecd74fe4fcdf"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "cd12c564bb0e13379bb01ac9937a707f"
server_key_iv: "ecad1dc9f2cf2de504c3834fc7c5d9f6"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.607365 22479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:14.623932 22479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:14.627524 22479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:14.627689 22479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:14.627948 22479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:14.628263 22479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:14.628298 22479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:14.628320 22479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:14.628367 22479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:14.634016 22479 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:34885
I20260924 07:22:14.634110 22607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:34885 every 8 connection(s)
I20260924 07:22:14.634399 22479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:22:14 dist-test-slave-rc1w krb5kdc[22104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:14.644490 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 22479
I20260924 07:22:14.644635 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:22:14.644833 4057 external_mini_cluster.cc:1428] Setting key e738ef4e9124391db19a30e3b9505a55
I20260924 07:22:14.645345 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.635070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:47611 (local address 127.3.246.126:34707)
0924 07:22:14.635138 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:14.635141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:14.635356 (+ 215us) server_negotiation.cc:407] Connection header received
0924 07:22:14.635878 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:14.635880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:14.635891 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:14.635952 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:14.636863 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.637374 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.638466 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.638683 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.638755 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.641704 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:14.641706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:14.641706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:14.641714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:14.643853 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:14.644290 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:14.644291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:14.644291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:14.644352 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:14.644612 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:14.644612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:14.644613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:14.644756 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:14.644819 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:14.645171 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 07:22:14.645259 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:22:14.645533 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.635049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:34707 (local address 127.3.246.67:47611)
0924 07:22:14.635236 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:14.635271 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:14.635763 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:14.636026 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:14.636030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:14.636589 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:14.636776 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.636781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.637471 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.637471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.638358 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.638361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.638767 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.638768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.638881 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.640000 (+ 1119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:14.640009 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:14.641576 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:14.643957 (+ 2381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:14.643964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:14.643966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:14.644197 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:14.644421 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:14.644422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:14.644423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:14.644537 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:14.644811 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:14.644817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:14.644993 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 07:22:14.645109 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 07:22:14.645276 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":110,"threads_started":1}
I20260924 07:22:14.645942 22608 heartbeater.cc:347] Connected to a master server at 127.3.246.126:34707
I20260924 07:22:14.646029 22608 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:14.646171 22608 heartbeater.cc:511] Master 127.3.246.126:34707 requested a full tablet report, sending...
I20260924 07:22:14.646483 22149 ts_manager.cc:194] Registered new tserver with Master: 833ac764dedc4f3d9215ecd74fe4fcdf (127.3.246.67:34885)
I20260924 07:22:14.646871 22149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:47611
I20260924 07:22:14.656816 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:22:14.666796 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.658214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.3.246.126:34707)
0924 07:22:14.658293 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:14.658296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:14.658379 (+ 83us) server_negotiation.cc:407] Connection header received
0924 07:22:14.658484 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:14.658485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:14.658496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:14.658573 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:14.659192 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.659734 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.660573 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.660794 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.660881 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.663056 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:14.663058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:14.663058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:14.663066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:14.665481 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:14.665954 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:14.665955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:14.665956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:14.666018 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:14.666246 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:14.666247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:14.666247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:14.666390 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:14.666506 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:14.666634 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 07:22:14.666698 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:22:14.670038 22149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:22:14.671706 22149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:22:14.678331 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.674109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:43803 (local address 127.0.0.1:54062)
0924 07:22:14.674281 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:14.674322 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:14.674390 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:14.674953 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:14.674954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:14.674956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:14.675002 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.675004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.676226 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.676227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.677328 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.677330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.678061 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.678062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.678123 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.678128 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:22:14.678189 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:22:14.678232 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":20,"thread_start_us":57,"threads_started":1}
I20260924 07:22:14.678520 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.674278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44864 (local address 127.3.246.67:34885)
0924 07:22:14.674535 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:14.674540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:14.674549 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:22:14.674613 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:14.674616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:14.674634 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:14.674719 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:14.674921 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.676076 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.677170 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.677872 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.678308 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.678365 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:14.678388 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 07:22:14.678428 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
I20260924 07:22:14.678668 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.674109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:34885 (local address 127.0.0.1:44864)
0924 07:22:14.674349 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:14.674360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:14.674414 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:14.674732 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:14.674735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:14.674737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:14.674788 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.674789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.676182 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.676183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.677083 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.677085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.678000 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.678001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.678079 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.678095 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 07:22:14.678146 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:22:14.678200 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260924 07:22:14.678737 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.674121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:37113 (local address 127.0.0.1:45480)
0924 07:22:14.674511 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:14.674520 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:14.674560 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:14.675046 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:14.675046 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:14.675049 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:14.675096 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.675098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.675966 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.675967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.677248 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.677251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.678173 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.678174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.678237 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.678240 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:22:14.678294 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 07:22:14.678327 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":27,"thread_start_us":94,"threads_started":1}
I20260924 07:22:14.678706 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.674348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.3.246.66:37113)
0924 07:22:14.674842 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:14.674846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:14.674856 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:22:14.674890 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:14.674892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:14.674906 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:14.674977 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:14.675182 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.675877 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.677340 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.678105 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.678171 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.678223 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:14.678322 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 07:22:14.678366 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":92,"threads_started":1}
I20260924 07:22:14.678958 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.674490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.3.246.65:43803)
0924 07:22:14.674692 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:14.674696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:14.674706 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:22:14.674742 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:14.674744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:14.674759 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:14.674840 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:14.675086 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.676151 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.677395 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.677957 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.678058 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.678119 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:14.678804 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 07:22:14.678864 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260924 07:22:14.679549 22274 tablet_service.cc:1467] Processing CreateTablet for tablet 71421bf546954bb6967b33496494cef0 (DEFAULT_TABLE table=test-table [id=c596834d489f4a0d829d37cacdf152d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:14.679584 22408 tablet_service.cc:1467] Processing CreateTablet for tablet 71421bf546954bb6967b33496494cef0 (DEFAULT_TABLE table=test-table [id=c596834d489f4a0d829d37cacdf152d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:14.679589 22542 tablet_service.cc:1467] Processing CreateTablet for tablet 71421bf546954bb6967b33496494cef0 (DEFAULT_TABLE table=test-table [id=c596834d489f4a0d829d37cacdf152d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:14.679904 22274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71421bf546954bb6967b33496494cef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:14.679906 22408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71421bf546954bb6967b33496494cef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:14.679906 22542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71421bf546954bb6967b33496494cef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:14.682474 22625 tablet_bootstrap.cc:492] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227: Bootstrap starting.
I20260924 07:22:14.683116 22626 tablet_bootstrap.cc:492] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747: Bootstrap starting.
I20260924 07:22:14.683393 22625 tablet_bootstrap.cc:654] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:14.683794 22626 tablet_bootstrap.cc:654] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:14.684015 22625 log.cc:824] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:14.684383 22626 log.cc:824] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:14.684705 22627 tablet_bootstrap.cc:492] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf: Bootstrap starting.
I20260924 07:22:14.684900 22625 tablet_bootstrap.cc:492] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227: No bootstrap required, opened a new log
I20260924 07:22:14.684976 22625 ts_tablet_manager.cc:1397] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:22:14.685187 22626 tablet_bootstrap.cc:492] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747: No bootstrap required, opened a new log
I20260924 07:22:14.685273 22626 ts_tablet_manager.cc:1397] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:22:14.685379 22627 tablet_bootstrap.cc:654] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:14.686012 22627 log.cc:824] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:14.686798 22627 tablet_bootstrap.cc:492] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf: No bootstrap required, opened a new log
I20260924 07:22:14.686870 22627 ts_tablet_manager.cc:1397] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:22:14.687201 22626 raft_consensus.cc:348] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bc3609370a4b7e8a5bc5b0ff6df227" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 43803 } } peers { permanent_uuid: "833ac764dedc4f3d9215ecd74fe4fcdf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34885 } } peers { permanent_uuid: "639c1854a1684fb782f717226d719747" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37113 } }
I20260924 07:22:14.687371 22626 raft_consensus.cc:374] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:14.687419 22626 raft_consensus.cc:729] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639c1854a1684fb782f717226d719747, State: Initialized, Role: FOLLOWER
I20260924 07:22:14.687414 22625 raft_consensus.cc:348] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bc3609370a4b7e8a5bc5b0ff6df227" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 43803 } } peers { permanent_uuid: "833ac764dedc4f3d9215ecd74fe4fcdf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34885 } } peers { permanent_uuid: "639c1854a1684fb782f717226d719747" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37113 } }
I20260924 07:22:14.687599 22625 raft_consensus.cc:374] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:14.687553 22626 consensus_queue.cc:260] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bc3609370a4b7e8a5bc5b0ff6df227" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 43803 } } peers { permanent_uuid: "833ac764dedc4f3d9215ecd74fe4fcdf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34885 } } peers { permanent_uuid: "639c1854a1684fb782f717226d719747" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37113 } }
I20260924 07:22:14.687645 22625 raft_consensus.cc:729] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bc3609370a4b7e8a5bc5b0ff6df227, State: Initialized, Role: FOLLOWER
I20260924 07:22:14.687777 22625 consensus_queue.cc:260] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bc3609370a4b7e8a5bc5b0ff6df227" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 43803 } } peers { permanent_uuid: "833ac764dedc4f3d9215ecd74fe4fcdf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34885 } } peers { permanent_uuid: "639c1854a1684fb782f717226d719747" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37113 } }
I20260924 07:22:14.687875 22474 heartbeater.cc:503] Master 127.3.246.126:34707 was elected leader, sending a full tablet report...
I20260924 07:22:14.688030 22625 ts_tablet_manager.cc:1428] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:22:14.688150 22340 heartbeater.cc:503] Master 127.3.246.126:34707 was elected leader, sending a full tablet report...
I20260924 07:22:14.687875 22626 ts_tablet_manager.cc:1428] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:22:14.691089 22627 raft_consensus.cc:348] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bc3609370a4b7e8a5bc5b0ff6df227" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 43803 } } peers { permanent_uuid: "833ac764dedc4f3d9215ecd74fe4fcdf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34885 } } peers { permanent_uuid: "639c1854a1684fb782f717226d719747" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37113 } }
I20260924 07:22:14.691339 22627 raft_consensus.cc:374] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:14.691393 22627 raft_consensus.cc:729] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833ac764dedc4f3d9215ecd74fe4fcdf, State: Initialized, Role: FOLLOWER
I20260924 07:22:14.691514 22627 consensus_queue.cc:260] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bc3609370a4b7e8a5bc5b0ff6df227" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 43803 } } peers { permanent_uuid: "833ac764dedc4f3d9215ecd74fe4fcdf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34885 } } peers { permanent_uuid: "639c1854a1684fb782f717226d719747" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37113 } }
I20260924 07:22:14.691836 22608 heartbeater.cc:503] Master 127.3.246.126:34707 was elected leader, sending a full tablet report...
I20260924 07:22:14.691820 22627 ts_tablet_manager.cc:1428] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260924 07:22:14.728917 22475 tablet.cc:2369] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:22:14.753065 22631 raft_consensus.cc:482] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:22:14.753196 22631 raft_consensus.cc:504] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bc3609370a4b7e8a5bc5b0ff6df227" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 43803 } } peers { permanent_uuid: "833ac764dedc4f3d9215ecd74fe4fcdf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34885 } } peers { permanent_uuid: "639c1854a1684fb782f717226d719747" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37113 } }
I20260924 07:22:14.753726 22631 leader_election.cc:288] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bc3609370a4b7e8a5bc5b0ff6df227 (127.3.246.65:43803), 833ac764dedc4f3d9215ecd74fe4fcdf (127.3.246.67:34885)
I20260924 07:22:14.757647 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.754003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:48673 (local address 127.3.246.67:34885)
0924 07:22:14.754067 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:14.754069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:14.754078 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:22:14.754113 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:14.754114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:14.754122 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:14.754178 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:14.754380 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.755146 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.756631 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.757228 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.757318 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.757328 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:14.757531 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 07:22:14.757574 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:22:14.757647 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.753966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:57213 (local address 127.3.246.65:43803)
0924 07:22:14.754013 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:14.754015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:14.754373 (+ 358us) server_negotiation.cc:407] Connection header received
0924 07:22:14.754405 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:14.754406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:14.754422 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:14.754489 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:14.754685 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.755341 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.756661 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.757233 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.757319 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.757328 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:14.757531 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 07:22:14.757575 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:22:14.757685 22476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.753885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:34885 (local address 127.3.246.66:48673)
0924 07:22:14.753932 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:14.753943 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:14.753986 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:14.754207 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:14.754208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:14.754209 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:14.754291 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.754293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.755241 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.755242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.756520 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.756523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.757342 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.757343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.757443 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.757445 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:22:14.757523 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:22:14.757561 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 07:22:14.757697 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.753887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:43803 (local address 127.3.246.66:57213)
0924 07:22:14.754251 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:14.754262 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:14.754318 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:14.754532 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:14.754533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:14.754536 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:14.754600 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.754601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.755425 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.755426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.756530 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:14.756532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.757339 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.757340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:14.757439 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.757444 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:22:14.757509 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:22:14.757555 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":185,"thread_start_us":113,"threads_started":1}
I20260924 07:22:14.758140 22562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71421bf546954bb6967b33496494cef0" candidate_uuid: "639c1854a1684fb782f717226d719747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833ac764dedc4f3d9215ecd74fe4fcdf" is_pre_election: true
I20260924 07:22:14.758306 22562 raft_consensus.cc:2454] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639c1854a1684fb782f717226d719747 in term 0.
I20260924 07:22:14.758283 22294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71421bf546954bb6967b33496494cef0" candidate_uuid: "639c1854a1684fb782f717226d719747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bc3609370a4b7e8a5bc5b0ff6df227" is_pre_election: true
I20260924 07:22:14.758430 22294 raft_consensus.cc:2454] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639c1854a1684fb782f717226d719747 in term 0.
I20260924 07:22:14.758615 22364 leader_election.cc:302] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639c1854a1684fb782f717226d719747, 833ac764dedc4f3d9215ecd74fe4fcdf; no voters:
I20260924 07:22:14.758733 22631 raft_consensus.cc:2781] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:22:14.758816 22631 raft_consensus.cc:482] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:22:14.758852 22631 raft_consensus.cc:3037] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:14.759593 22631 raft_consensus.cc:504] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bc3609370a4b7e8a5bc5b0ff6df227" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 43803 } } peers { permanent_uuid: "833ac764dedc4f3d9215ecd74fe4fcdf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34885 } } peers { permanent_uuid: "639c1854a1684fb782f717226d719747" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37113 } }
I20260924 07:22:14.759740 22631 leader_election.cc:288] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [CANDIDATE]: Term 1 election: Requested vote from peers 43bc3609370a4b7e8a5bc5b0ff6df227 (127.3.246.65:43803), 833ac764dedc4f3d9215ecd74fe4fcdf (127.3.246.67:34885)
I20260924 07:22:14.759902 22294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71421bf546954bb6967b33496494cef0" candidate_uuid: "639c1854a1684fb782f717226d719747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bc3609370a4b7e8a5bc5b0ff6df227"
I20260924 07:22:14.759927 22562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71421bf546954bb6967b33496494cef0" candidate_uuid: "639c1854a1684fb782f717226d719747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833ac764dedc4f3d9215ecd74fe4fcdf"
I20260924 07:22:14.759994 22562 raft_consensus.cc:3037] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:14.760010 22294 raft_consensus.cc:3037] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:14.760703 22562 raft_consensus.cc:2454] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639c1854a1684fb782f717226d719747 in term 1.
I20260924 07:22:14.760838 22294 raft_consensus.cc:2454] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639c1854a1684fb782f717226d719747 in term 1.
I20260924 07:22:14.760895 22364 leader_election.cc:302] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639c1854a1684fb782f717226d719747, 833ac764dedc4f3d9215ecd74fe4fcdf; no voters:
I20260924 07:22:14.760993 22631 raft_consensus.cc:2781] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:14.761169 22631 raft_consensus.cc:686] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [term 1 LEADER]: Becoming Leader. State: Replica: 639c1854a1684fb782f717226d719747, State: Running, Role: LEADER
I20260924 07:22:14.761286 22631 consensus_queue.cc:237] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bc3609370a4b7e8a5bc5b0ff6df227" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 43803 } } peers { permanent_uuid: "833ac764dedc4f3d9215ecd74fe4fcdf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34885 } } peers { permanent_uuid: "639c1854a1684fb782f717226d719747" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37113 } }
I20260924 07:22:14.762166 22148 catalog_manager.cc:5495] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 reported cstate change: term changed from 0 to 1, leader changed from <none> to 639c1854a1684fb782f717226d719747 (127.3.246.66). New cstate: current_term: 1 leader_uuid: "639c1854a1684fb782f717226d719747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bc3609370a4b7e8a5bc5b0ff6df227" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 43803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "833ac764dedc4f3d9215ecd74fe4fcdf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639c1854a1684fb782f717226d719747" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 37113 } health_report { overall_health: HEALTHY } } }
I20260924 07:22:14.775460 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.773019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.3.246.66:37113)
0924 07:22:14.773070 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:14.773073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:14.773082 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:22:14.773130 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:14.773131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:14.773148 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:14.773215 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:14.773421 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.773962 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.774723 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.774911 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.774975 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.775177 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:14.775262 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:14.775330 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:14.775381 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 07:22:14.775389 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:22:14.777271 22562 raft_consensus.cc:1261] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf [term 1 FOLLOWER]: Refusing update from remote peer 639c1854a1684fb782f717226d719747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:14.777278 22294 raft_consensus.cc:1261] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227 [term 1 FOLLOWER]: Refusing update from remote peer 639c1854a1684fb782f717226d719747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:14.777681 22635 consensus_queue.cc:1035] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bc3609370a4b7e8a5bc5b0ff6df227" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 43803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:14.777827 22631 consensus_queue.cc:1035] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833ac764dedc4f3d9215ecd74fe4fcdf" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 34885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:14.780108 22638 mvcc.cc:205] Tried to move back new op lower bound from 7332800654444765184 to 7332800654382821376. Current Snapshot: MvccSnapshot[applied={T|T < 7332800654444765184}]
I20260924 07:22:14.780517 22639 mvcc.cc:205] Tried to move back new op lower bound from 7332800654444765184 to 7332800654382821376. Current Snapshot: MvccSnapshot[applied={T|T < 7332800654444765184}]
I20260924 07:22:14.780875 22640 mvcc.cc:205] Tried to move back new op lower bound from 7332800654444765184 to 7332800654382821376. Current Snapshot: MvccSnapshot[applied={T|T < 7332800654444765184}]
I20260924 07:22:14.782789 22148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:22:14.782888 22148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:22:14.783633 22148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82a1b4285890438e8350e6fb6385d9c2: Sending DeleteTablet for 3 replicas of tablet 71421bf546954bb6967b33496494cef0
I20260924 07:22:14.783957 22274 tablet_service.cc:1514] Processing DeleteTablet for tablet 71421bf546954bb6967b33496494cef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:14 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:54062
I20260924 07:22:14.783965 22408 tablet_service.cc:1514] Processing DeleteTablet for tablet 71421bf546954bb6967b33496494cef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:14 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:45480
I20260924 07:22:14.784133 22542 tablet_service.cc:1514] Processing DeleteTablet for tablet 71421bf546954bb6967b33496494cef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:14 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:44864
I20260924 07:22:14.784320 22649 tablet_replica.cc:330] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf: stopping tablet replica
I20260924 07:22:14.784444 22649 raft_consensus.cc:2229] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:22:14.784441 22647 tablet_replica.cc:330] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227: stopping tablet replica
I20260924 07:22:14.784524 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 22211
I20260924 07:22:14.784536 22647 raft_consensus.cc:2229] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:22:14.784642 22647 raft_consensus.cc:2258] T 71421bf546954bb6967b33496494cef0 P 43bc3609370a4b7e8a5bc5b0ff6df227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:14.784790 22648 tablet_replica.cc:330] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747: stopping tablet replica
I20260924 07:22:14.784871 22648 raft_consensus.cc:2229] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:22:14.784953 22648 raft_consensus.cc:2258] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:14.784559 22649 raft_consensus.cc:2258] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:14.785742 22648 ts_tablet_manager.cc:1905] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:22:14.785892 22649 ts_tablet_manager.cc:1905] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:22:14.787326 22649 ts_tablet_manager.cc:1918] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:22:14.787418 22649 log.cc:1196] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/71421bf546954bb6967b33496494cef0
I20260924 07:22:14.787693 22649 ts_tablet_manager.cc:1939] T 71421bf546954bb6967b33496494cef0 P 833ac764dedc4f3d9215ecd74fe4fcdf: Deleting consensus metadata
I20260924 07:22:14.787864 22648 ts_tablet_manager.cc:1918] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:22:14.787942 22648 log.cc:1196] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/71421bf546954bb6967b33496494cef0
I20260924 07:22:14.788103 22136 catalog_manager.cc:4841] TS 833ac764dedc4f3d9215ecd74fe4fcdf (127.3.246.67:34885): tablet 71421bf546954bb6967b33496494cef0 (table test-table [id=c596834d489f4a0d829d37cacdf152d0]) successfully deleted
I20260924 07:22:14.788200 22648 ts_tablet_manager.cc:1939] T 71421bf546954bb6967b33496494cef0 P 639c1854a1684fb782f717226d719747: Deleting consensus metadata
I20260924 07:22:14.788610 22135 catalog_manager.cc:4841] TS 639c1854a1684fb782f717226d719747 (127.3.246.66:37113): tablet 71421bf546954bb6967b33496494cef0 (table test-table [id=c596834d489f4a0d829d37cacdf152d0]) successfully deleted
W20260924 07:22:14.791664 22134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:43803 (error 108)
I20260924 07:22:14.792191 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.792045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:43803 (local address 127.0.0.1:54064)
0924 07:22:14.792099 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:14.792135 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 07:22:14.792258 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 22345
W20260924 07:22:14.796831 22134 catalog_manager.cc:4584] TS 43bc3609370a4b7e8a5bc5b0ff6df227 (127.3.246.65:43803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71421bf546954bb6967b33496494cef0: Network error: Client connection negotiation failed: client connection to 127.3.246.65:43803: connect: Connection refused (error 111)
I20260924 07:22:14.796932 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 22479
I20260924 07:22:14.801431 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 22117
2026-09-24T07:22:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1948 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T07:22:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:14Z Disabled control of system clock
I20260924 07:22:14.829059 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:38375
--webserver_interface=127.3.246.126
--webserver_port=0
--builtin_ntp_servers=127.3.246.84:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:38375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:22:14.909140 22654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:14.909354 22654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:14.909374 22654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:14.910593 22654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:14.910633 22654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:14.910646 22654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:14.910658 22654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:14.910670 22654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:14.912017 22654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:38375
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:38375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:14.912283 22654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:14.912581 22654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:14.917338 22659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:14.917341 22660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:14.917358 22662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:14.917757 22654 server_base.cc:1034] running on GCE node
I20260924 07:22:14.918028 22654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:14.918323 22654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:14.919485 22654 hybrid_clock.cc:648] HybridClock initialized: now 1790234534919472 us; error 44 us; skew 500 ppm
I20260924 07:22:14.921741 22654 webserver.cc:460] Webserver started at http://127.3.246.126:38107/ using document root <none> and password file <none>
I20260924 07:22:14.921979 22654 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:14.922046 22654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:14.922183 22654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:14.923303 22654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "ca831585d7014381adf52dfffe54f68e"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "e3caa185026a6ff07efe6284f5234948"
server_key_iv: "3f6df915562ba5c2ce55ec33031a6be4"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.923601 22654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "ca831585d7014381adf52dfffe54f68e"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "e3caa185026a6ff07efe6284f5234948"
server_key_iv: "3f6df915562ba5c2ce55ec33031a6be4"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.924763 22654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:22:14.925371 22668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:14.925562 22654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:14.925661 22654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "ca831585d7014381adf52dfffe54f68e"
format_stamp: "Formatted at 2026-09-24 07:22:14 on dist-test-slave-rc1w"
server_key: "e3caa185026a6ff07efe6284f5234948"
server_key_iv: "3f6df915562ba5c2ce55ec33031a6be4"
server_key_version: "encryptionkey@0"
I20260924 07:22:14.925781 22654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:14.948956 22654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:14.949226 22654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:14.949335 22654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:14.953110 22654 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:38375
I20260924 07:22:14.953183 22720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:38375 every 8 connection(s)
I20260924 07:22:14.953516 22654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:14.953677 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 22654
I20260924 07:22:14.953816 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:14.954056 4057 external_mini_cluster.cc:1428] Setting key c9e08baf284045da54d448aedf096362
I20260924 07:22:14.954728 22721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:14.957753 22721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e: Bootstrap starting.
I20260924 07:22:14.958420 22721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:14.958766 22721 log.cc:824] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:14.959434 22721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e: No bootstrap required, opened a new log
I20260924 07:22:14.960305 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:14.954621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.3.246.126:38375)
0924 07:22:14.954860 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:14.954867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:14.954893 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:22:14.955329 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:14.955335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:14.955398 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:14.955509 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:14.955762 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.956599 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.957731 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:14.958168 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:14.958256 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:14.959701 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:14.959705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:14.959705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:14.959852 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:14.959883 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:14.959894 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:14.959970 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:14.960047 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 07:22:14.960128 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":104,"threads_started":1}
I20260924 07:22:14.961445 22721 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca831585d7014381adf52dfffe54f68e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 38375 } }
I20260924 07:22:14.961598 22721 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:14.961647 22721 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca831585d7014381adf52dfffe54f68e, State: Initialized, Role: FOLLOWER
I20260924 07:22:14.961804 22721 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca831585d7014381adf52dfffe54f68e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 38375 } }
I20260924 07:22:14.961887 22721 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:14.961930 22721 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:14.961978 22721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:14.962513 22721 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca831585d7014381adf52dfffe54f68e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 38375 } }
I20260924 07:22:14.962630 22721 leader_election.cc:302] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca831585d7014381adf52dfffe54f68e; no voters:
I20260924 07:22:14.962814 22721 leader_election.cc:288] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:14.962865 22726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:14.963025 22726 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [term 1 LEADER]: Becoming Leader. State: Replica: ca831585d7014381adf52dfffe54f68e, State: Running, Role: LEADER
I20260924 07:22:14.963059 22721 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:14.963114 22726 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca831585d7014381adf52dfffe54f68e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 38375 } }
I20260924 07:22:14.963477 22728 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca831585d7014381adf52dfffe54f68e. Latest consensus state: current_term: 1 leader_uuid: "ca831585d7014381adf52dfffe54f68e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca831585d7014381adf52dfffe54f68e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 38375 } } }
I20260924 07:22:14.963474 22727 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca831585d7014381adf52dfffe54f68e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca831585d7014381adf52dfffe54f68e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 38375 } } }
I20260924 07:22:14.963581 22728 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:14.963613 22727 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:14.963780 22733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:14.964309 22733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:14.965909 22733 catalog_manager.cc:1347] Generated new cluster ID: 6d42c1632e4d44f9ac710727a29a0c9c
I20260924 07:22:14.965965 22733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:14.978474 22733 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:14.979251 22733 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:14.983278 22733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e: Generated new TSK 0
I20260924 07:22:14.983484 22733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:14.986620 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:38375
--builtin_ntp_servers=127.3.246.84:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:22:15.066658 22745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:15.066848 22745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:15.066870 22745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:15.068012 22745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:15.068053 22745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:15.068077 22745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:22:15.069391 22745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:38375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:15.069619 22745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:15.069931 22745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:15.075575 22750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:15.075592 22751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:15.075621 22753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:15.075863 22745 server_base.cc:1034] running on GCE node
I20260924 07:22:15.076066 22745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:15.076308 22745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:15.077474 22745 hybrid_clock.cc:648] HybridClock initialized: now 1790234535077457 us; error 33 us; skew 500 ppm
I20260924 07:22:15.088883 22745 webserver.cc:460] Webserver started at http://127.3.246.65:41203/ using document root <none> and password file <none>
I20260924 07:22:15.089206 22745 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:15.089284 22745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:15.089442 22745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:15.091059 22745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "999e3a35b4d4491b9b1cd70054206eec"
format_stamp: "Formatted at 2026-09-24 07:22:15 on dist-test-slave-rc1w"
server_key: "4d05171c16100ff4069b889706a61e89"
server_key_iv: "332d1325052f6006c4bb1aa34ef9965d"
server_key_version: "encryptionkey@0"
I20260924 07:22:15.091526 22745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "999e3a35b4d4491b9b1cd70054206eec"
format_stamp: "Formatted at 2026-09-24 07:22:15 on dist-test-slave-rc1w"
server_key: "4d05171c16100ff4069b889706a61e89"
server_key_iv: "332d1325052f6006c4bb1aa34ef9965d"
server_key_version: "encryptionkey@0"
I20260924 07:22:15.092983 22745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:22:15.093972 22759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:15.094203 22745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:22:15.094317 22745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "999e3a35b4d4491b9b1cd70054206eec"
format_stamp: "Formatted at 2026-09-24 07:22:15 on dist-test-slave-rc1w"
server_key: "4d05171c16100ff4069b889706a61e89"
server_key_iv: "332d1325052f6006c4bb1aa34ef9965d"
server_key_version: "encryptionkey@0"
I20260924 07:22:15.094403 22745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:15.115550 22745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:15.115973 22745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:15.116140 22745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:15.116387 22745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:15.116724 22745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:15.116780 22745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:15.116822 22745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:15.116851 22745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:15.122692 22745 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:34343
I20260924 07:22:15.122731 22872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:34343 every 8 connection(s)
I20260924 07:22:15.123131 22745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:22:15.129110 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.123852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:33619 (local address 127.3.246.126:38375)
0924 07:22:15.123922 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:15.123925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:15.124188 (+ 263us) server_negotiation.cc:407] Connection header received
0924 07:22:15.124720 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:15.124721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:15.124736 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:15.124799 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:15.125525 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.126361 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.127545 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.127780 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.127845 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.128798 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:15.128799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:15.128800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:15.128806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:15.128823 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:15.128826 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:15.128875 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:15.128974 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 07:22:15.129015 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:22:15.129290 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.123851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:38375 (local address 127.3.246.65:33619)
0924 07:22:15.124068 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:15.124111 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:15.124603 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:15.124865 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:15.124869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:15.124881 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:15.125412 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.125420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.126479 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.126480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.127436 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.127439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.127903 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.127904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.128061 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.128651 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:22:15.128652 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:15.128687 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:22:15.128709 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:15.128875 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:15.128880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:15.128883 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:22:15.128974 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 07:22:15.129072 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260924 07:22:15.136786 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 22745
I20260924 07:22:15.136906 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:22:15.137146 4057 external_mini_cluster.cc:1428] Setting key 672f3d363c3a25de2cb1a2bd2c8c34a3
I20260924 07:22:15.138028 22873 heartbeater.cc:347] Connected to a master server at 127.3.246.126:38375
I20260924 07:22:15.138144 22873 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:15.138219 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:38375
--builtin_ntp_servers=127.3.246.84:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:22:15.138366 22873 heartbeater.cc:511] Master 127.3.246.126:38375 requested a full tablet report, sending...
I20260924 07:22:15.138948 22685 ts_manager.cc:194] Registered new tserver with Master: 999e3a35b4d4491b9b1cd70054206eec (127.3.246.65:34343)
I20260924 07:22:15.139895 22685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.246.65:33619
W20260924 07:22:15.236815 22876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:15.237114 22876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:15.237157 22876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:15.238396 22876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:15.238459 22876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:15.238517 22876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:22:15.239854 22876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:38375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:15.240118 22876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:15.240407 22876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:15.245265 22884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:15.245292 22882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:15.245271 22881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:15.245476 22876 server_base.cc:1034] running on GCE node
I20260924 07:22:15.245762 22876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:15.246038 22876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:15.247198 22876 hybrid_clock.cc:648] HybridClock initialized: now 1790234535247195 us; error 35 us; skew 500 ppm
I20260924 07:22:15.248430 22876 webserver.cc:460] Webserver started at http://127.3.246.66:40507/ using document root <none> and password file <none>
I20260924 07:22:15.248678 22876 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:15.248749 22876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:15.248883 22876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:15.250007 22876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756"
format_stamp: "Formatted at 2026-09-24 07:22:15 on dist-test-slave-rc1w"
server_key: "4774171ef78bba1d86bbc7f384c1b622"
server_key_iv: "ff1c73b341304e9cfd0ef6843c883a7d"
server_key_version: "encryptionkey@0"
I20260924 07:22:15.250365 22876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756"
format_stamp: "Formatted at 2026-09-24 07:22:15 on dist-test-slave-rc1w"
server_key: "4774171ef78bba1d86bbc7f384c1b622"
server_key_iv: "ff1c73b341304e9cfd0ef6843c883a7d"
server_key_version: "encryptionkey@0"
I20260924 07:22:15.251701 22876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:22:15.252462 22890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:15.252638 22876 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:22:15.252735 22876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756"
format_stamp: "Formatted at 2026-09-24 07:22:15 on dist-test-slave-rc1w"
server_key: "4774171ef78bba1d86bbc7f384c1b622"
server_key_iv: "ff1c73b341304e9cfd0ef6843c883a7d"
server_key_version: "encryptionkey@0"
I20260924 07:22:15.252820 22876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:15.269098 22876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:15.269331 22876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:15.269428 22876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:15.269587 22876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:15.269928 22876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:15.270020 22876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:15.270062 22876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:15.270084 22876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:15.275494 22876 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:39135
I20260924 07:22:15.275563 23003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:39135 every 8 connection(s)
I20260924 07:22:15.275882 22876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:22:15.280903 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.276613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:54211 (local address 127.3.246.126:38375)
0924 07:22:15.276678 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:15.276680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:15.276897 (+ 217us) server_negotiation.cc:407] Connection header received
0924 07:22:15.277350 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:15.277351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:15.277360 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:15.277408 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:15.278093 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.278579 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.279565 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.279807 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.279888 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.280658 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:15.280659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:15.280660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:15.280664 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:15.280679 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:15.280683 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:15.280744 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:15.280810 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 07:22:15.280840 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:22:15.281049 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.276602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:38375 (local address 127.3.246.66:54211)
0924 07:22:15.276786 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:15.276817 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:15.277273 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:15.277466 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:15.277468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:15.277476 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:15.278021 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.278026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.278660 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.278661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.279432 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.279434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.279900 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.279901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.280010 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.280511 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:22:15.280511 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:15.280538 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:22:15.280559 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:15.280746 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:15.280748 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:15.280750 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:22:15.280797 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 07:22:15.280869 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260924 07:22:15.281738 23004 heartbeater.cc:347] Connected to a master server at 127.3.246.126:38375
I20260924 07:22:15.281814 23004 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:15.281945 23004 heartbeater.cc:511] Master 127.3.246.126:38375 requested a full tablet report, sending...
I20260924 07:22:15.282276 22685 ts_manager.cc:194] Registered new tserver with Master: 99d41d1a8bcd4c4bb0cd2ea8d1a31756 (127.3.246.66:39135)
I20260924 07:22:15.282676 22685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.246.66:54211
I20260924 07:22:15.283699 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 22876
I20260924 07:22:15.283766 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:22:15.283905 4057 external_mini_cluster.cc:1428] Setting key 6d5e3d34dda19037ac91edd9aeeb9c08
I20260924 07:22:15.284759 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:38375
--builtin_ntp_servers=127.3.246.84:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:22:15.363459 23007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:15.363739 23007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:15.363775 23007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:15.364953 23007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:15.365010 23007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:15.365051 23007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:22:15.366395 23007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:38375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:15.366640 23007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:15.366925 23007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:15.371848 23013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:15.371996 23015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:15.371866 23012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:15.372087 23007 server_base.cc:1034] running on GCE node
I20260924 07:22:15.372287 23007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:15.372524 23007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:15.373759 23007 hybrid_clock.cc:648] HybridClock initialized: now 1790234535373729 us; error 48 us; skew 500 ppm
I20260924 07:22:15.375064 23007 webserver.cc:460] Webserver started at http://127.3.246.67:44847/ using document root <none> and password file <none>
I20260924 07:22:15.375332 23007 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:15.375403 23007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:15.375545 23007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:15.376655 23007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "a8b3ff4484044a7d8eb4ec1983eeed99"
format_stamp: "Formatted at 2026-09-24 07:22:15 on dist-test-slave-rc1w"
server_key: "8b7c226b82ef008e49bb902d9d01751a"
server_key_iv: "bf9e7ab0b59dfd5ae42432fc1699ab96"
server_key_version: "encryptionkey@0"
I20260924 07:22:15.377017 23007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "a8b3ff4484044a7d8eb4ec1983eeed99"
format_stamp: "Formatted at 2026-09-24 07:22:15 on dist-test-slave-rc1w"
server_key: "8b7c226b82ef008e49bb902d9d01751a"
server_key_iv: "bf9e7ab0b59dfd5ae42432fc1699ab96"
server_key_version: "encryptionkey@0"
I20260924 07:22:15.378418 23007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:22:15.379297 23021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:15.379478 23007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:22:15.379580 23007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "a8b3ff4484044a7d8eb4ec1983eeed99"
format_stamp: "Formatted at 2026-09-24 07:22:15 on dist-test-slave-rc1w"
server_key: "8b7c226b82ef008e49bb902d9d01751a"
server_key_iv: "bf9e7ab0b59dfd5ae42432fc1699ab96"
server_key_version: "encryptionkey@0"
I20260924 07:22:15.379689 23007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:15.391599 23007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:15.391855 23007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:15.391961 23007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:15.392128 23007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:15.392470 23007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:15.392527 23007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:15.392570 23007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:15.392599 23007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:15.399197 23007 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:40707
I20260924 07:22:15.399235 23134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:40707 every 8 connection(s)
I20260924 07:22:15.399583 23007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:22:15.404672 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.400449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:56005 (local address 127.3.246.126:38375)
0924 07:22:15.400652 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:15.400654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:15.400711 (+ 57us) server_negotiation.cc:407] Connection header received
0924 07:22:15.401187 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:15.401188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:15.401199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:15.401263 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:15.402021 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.402523 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.403483 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.403665 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.403717 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.404451 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:15.404452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:15.404452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:15.404456 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:15.404470 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:15.404473 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:15.404516 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:15.404594 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 07:22:15.404615 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:22:15.404915 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.400406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:38375 (local address 127.3.246.67:56005)
0924 07:22:15.400585 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:15.400634 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:15.401103 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:15.401335 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:15.401339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:15.401346 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:15.401915 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.401921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.402614 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.402615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.403406 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.403409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.403738 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.403739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.403844 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.404327 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:22:15.404327 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:15.404354 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:22:15.404375 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:15.404527 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:15.404531 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:15.404533 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:22:15.404584 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:22:15.404667 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260924 07:22:15.405450 23135 heartbeater.cc:347] Connected to a master server at 127.3.246.126:38375
I20260924 07:22:15.405557 23135 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:15.405797 23135 heartbeater.cc:511] Master 127.3.246.126:38375 requested a full tablet report, sending...
I20260924 07:22:15.406193 22685 ts_manager.cc:194] Registered new tserver with Master: a8b3ff4484044a7d8eb4ec1983eeed99 (127.3.246.67:40707)
I20260924 07:22:15.406513 22685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.246.67:56005
I20260924 07:22:15.409263 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 23007
I20260924 07:22:15.409361 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:22:15.409551 4057 external_mini_cluster.cc:1428] Setting key a1560841a8c52aa46391ba07b72b5f30
I20260924 07:22:15.410379 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:22:15.414453 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.411680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.3.246.126:38375)
0924 07:22:15.411713 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:15.411715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:15.411870 (+ 155us) server_negotiation.cc:407] Connection header received
0924 07:22:15.411916 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:15.411917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:15.411925 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:15.411972 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:15.412120 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.412549 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.413329 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.413506 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.413568 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.414250 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:15.414252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:15.414252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:15.414255 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:15.414267 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:15.414270 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:15.414335 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:15.414394 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 07:22:15.414402 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 07:22:15.415855 22685 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:22:15.417361 22685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:22:15.423336 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.419413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:34343 (local address 127.0.0.1:58882)
0924 07:22:15.419583 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:15.419632 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:15.419695 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:15.419948 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:15.419950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:15.419952 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:15.420014 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.420016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.421021 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.421022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.422178 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.422181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.423055 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.423056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.423128 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.423141 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 07:22:15.423215 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:22:15.423266 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":65,"threads_started":1}
I20260924 07:22:15.423349 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.419587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.3.246.65:34343)
0924 07:22:15.419792 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:15.419803 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:15.419811 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:22:15.419842 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:15.419843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:15.419855 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:15.419942 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:15.420101 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.420934 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.422277 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.422954 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.423052 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.423153 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:15.423229 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 07:22:15.423266 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
I20260924 07:22:15.423620 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.419413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:40707 (local address 127.0.0.1:43342)
0924 07:22:15.419882 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:15.419891 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:15.419964 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:15.420221 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:15.420222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:15.420224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:15.420267 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.420268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.421528 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.421529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.422510 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.422512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.423394 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.423395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.423455 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.423459 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:15.423493 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 07:22:15.423535 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":4,"thread_start_us":80,"threads_started":1}
I20260924 07:22:15.423902 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.419434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:39135 (local address 127.0.0.1:42740)
0924 07:22:15.420032 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:15.420042 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:15.420083 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:15.420325 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:15.420326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:15.420328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:15.420357 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.420357 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.421574 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.421575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.422716 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.422718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.423586 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.423587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.423654 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.423658 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:15.423776 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 07:22:15.423823 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":45,"thread_start_us":99,"threads_started":1}
I20260924 07:22:15.423943 23147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.419665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.3.246.67:40707)
0924 07:22:15.419826 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:15.419830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:15.419954 (+ 124us) server_negotiation.cc:407] Connection header received
0924 07:22:15.420042 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:15.420044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:15.420060 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:15.420137 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:15.420341 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.421422 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.422634 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.423285 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.423727 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.423813 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:15.423837 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 07:22:15.423871 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260924 07:22:15.424111 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.419667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.3.246.66:39135)
0924 07:22:15.420116 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:15.420120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:15.420130 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:22:15.420167 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:15.420169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:15.420184 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:15.420255 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:15.420432 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.421499 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.422797 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.423500 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.423573 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.423661 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:15.423903 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 07:22:15.423945 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":82,"threads_started":1}
I20260924 07:22:15.424440 22807 tablet_service.cc:1467] Processing CreateTablet for tablet f881088faaa5402e8fb96aabbf368d78 (DEFAULT_TABLE table=test-table [id=a3d878c84d2c40388bd7305c4db3a602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:15.424566 23069 tablet_service.cc:1467] Processing CreateTablet for tablet f881088faaa5402e8fb96aabbf368d78 (DEFAULT_TABLE table=test-table [id=a3d878c84d2c40388bd7305c4db3a602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:15.424806 23069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f881088faaa5402e8fb96aabbf368d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:15.424798 22807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f881088faaa5402e8fb96aabbf368d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:15.424743 22938 tablet_service.cc:1467] Processing CreateTablet for tablet f881088faaa5402e8fb96aabbf368d78 (DEFAULT_TABLE table=test-table [id=a3d878c84d2c40388bd7305c4db3a602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:15.425030 22938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f881088faaa5402e8fb96aabbf368d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:15.427835 23149 tablet_bootstrap.cc:492] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99: Bootstrap starting.
I20260924 07:22:15.428246 23151 tablet_bootstrap.cc:492] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec: Bootstrap starting.
I20260924 07:22:15.428525 23149 tablet_bootstrap.cc:654] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:15.428884 23151 tablet_bootstrap.cc:654] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:15.428925 23149 log.cc:824] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:15.429219 23151 log.cc:824] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:15.429543 23150 tablet_bootstrap.cc:492] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756: Bootstrap starting.
I20260924 07:22:15.429670 23149 tablet_bootstrap.cc:492] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99: No bootstrap required, opened a new log
I20260924 07:22:15.429764 23149 ts_tablet_manager.cc:1397] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:22:15.429960 23151 tablet_bootstrap.cc:492] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec: No bootstrap required, opened a new log
I20260924 07:22:15.430027 23151 ts_tablet_manager.cc:1397] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:22:15.430212 23150 tablet_bootstrap.cc:654] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:15.430521 23150 log.cc:824] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:15.431298 23150 tablet_bootstrap.cc:492] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756: No bootstrap required, opened a new log
I20260924 07:22:15.431366 23150 ts_tablet_manager.cc:1397] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:22:15.431932 23149 raft_consensus.cc:348] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999e3a35b4d4491b9b1cd70054206eec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34343 } } peers { permanent_uuid: "a8b3ff4484044a7d8eb4ec1983eeed99" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40707 } } peers { permanent_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 39135 } }
I20260924 07:22:15.432078 23149 raft_consensus.cc:374] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:15.432122 23149 raft_consensus.cc:729] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b3ff4484044a7d8eb4ec1983eeed99, State: Initialized, Role: FOLLOWER
I20260924 07:22:15.432235 23149 consensus_queue.cc:260] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999e3a35b4d4491b9b1cd70054206eec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34343 } } peers { permanent_uuid: "a8b3ff4484044a7d8eb4ec1983eeed99" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40707 } } peers { permanent_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 39135 } }
I20260924 07:22:15.432492 23149 ts_tablet_manager.cc:1428] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 07:22:15.432618 23135 heartbeater.cc:503] Master 127.3.246.126:38375 was elected leader, sending a full tablet report...
I20260924 07:22:15.432660 23151 raft_consensus.cc:348] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999e3a35b4d4491b9b1cd70054206eec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34343 } } peers { permanent_uuid: "a8b3ff4484044a7d8eb4ec1983eeed99" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40707 } } peers { permanent_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 39135 } }
I20260924 07:22:15.432821 23151 raft_consensus.cc:374] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:15.432865 23151 raft_consensus.cc:729] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999e3a35b4d4491b9b1cd70054206eec, State: Initialized, Role: FOLLOWER
I20260924 07:22:15.432972 23151 consensus_queue.cc:260] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999e3a35b4d4491b9b1cd70054206eec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34343 } } peers { permanent_uuid: "a8b3ff4484044a7d8eb4ec1983eeed99" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40707 } } peers { permanent_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 39135 } }
I20260924 07:22:15.433225 23151 ts_tablet_manager.cc:1428] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 07:22:15.433280 22873 heartbeater.cc:503] Master 127.3.246.126:38375 was elected leader, sending a full tablet report...
I20260924 07:22:15.433928 23150 raft_consensus.cc:348] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999e3a35b4d4491b9b1cd70054206eec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34343 } } peers { permanent_uuid: "a8b3ff4484044a7d8eb4ec1983eeed99" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40707 } } peers { permanent_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 39135 } }
I20260924 07:22:15.434079 23150 raft_consensus.cc:374] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:15.434139 23150 raft_consensus.cc:729] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d41d1a8bcd4c4bb0cd2ea8d1a31756, State: Initialized, Role: FOLLOWER
I20260924 07:22:15.434250 23150 consensus_queue.cc:260] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999e3a35b4d4491b9b1cd70054206eec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34343 } } peers { permanent_uuid: "a8b3ff4484044a7d8eb4ec1983eeed99" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40707 } } peers { permanent_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 39135 } }
I20260924 07:22:15.434494 23150 ts_tablet_manager.cc:1428] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:22:15.434572 23004 heartbeater.cc:503] Master 127.3.246.126:38375 was elected leader, sending a full tablet report...
W20260924 07:22:15.526433 23005 tablet.cc:2369] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:22:15.534893 23157 raft_consensus.cc:482] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:22:15.535017 23157 raft_consensus.cc:504] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999e3a35b4d4491b9b1cd70054206eec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34343 } } peers { permanent_uuid: "a8b3ff4484044a7d8eb4ec1983eeed99" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40707 } } peers { permanent_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 39135 } }
I20260924 07:22:15.535769 23157 leader_election.cc:288] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 999e3a35b4d4491b9b1cd70054206eec (127.3.246.65:34343), a8b3ff4484044a7d8eb4ec1983eeed99 (127.3.246.67:40707)
I20260924 07:22:15.538897 23147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.535738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:42009 (local address 127.3.246.67:40707)
0924 07:22:15.535785 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:15.535787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:15.535797 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:22:15.535828 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:15.535829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:15.535841 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:15.535895 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:15.536107 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.536618 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.538037 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.538579 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.538650 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.538659 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:15.538793 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 07:22:15.538830 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:22:15.538897 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.535589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:40707 (local address 127.3.246.66:42009)
0924 07:22:15.535639 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:15.535650 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:15.535689 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:15.535947 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:15.535948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:15.535950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:15.536022 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.536024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.536711 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.536712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.537943 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.537946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.538650 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.538651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.538726 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.538730 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:15.538782 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 07:22:15.538828 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 07:22:15.539227 23089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f881088faaa5402e8fb96aabbf368d78" candidate_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8b3ff4484044a7d8eb4ec1983eeed99" is_pre_election: true
I20260924 07:22:15.539400 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.535627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:34343 (local address 127.3.246.66:50543)
0924 07:22:15.535811 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:15.535820 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:15.535872 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:15.536025 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:15.536026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:15.536027 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:15.536125 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.536127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.536776 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.536777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.538145 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:15.538148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.538922 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.538924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:15.539158 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.539163 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:22:15.539251 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:22:15.539289 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":239,"thread_start_us":114,"threads_started":1}
I20260924 07:22:15.539418 23089 raft_consensus.cc:2454] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99d41d1a8bcd4c4bb0cd2ea8d1a31756 in term 0.
I20260924 07:22:15.539640 22894 leader_election.cc:302] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99d41d1a8bcd4c4bb0cd2ea8d1a31756, a8b3ff4484044a7d8eb4ec1983eeed99; no voters:
I20260924 07:22:15.539762 23157 raft_consensus.cc:2781] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:22:15.539800 23157 raft_consensus.cc:482] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:22:15.539820 23157 raft_consensus.cc:3037] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:15.539821 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.535856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:50543 (local address 127.3.246.65:34343)
0924 07:22:15.535902 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:15.535904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:15.535912 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:22:15.535942 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:15.535943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:15.535951 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:15.536012 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:15.536212 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.536698 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.538236 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.538790 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.538867 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.538877 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:15.539337 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 07:22:15.539752 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:22:15.540205 22827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f881088faaa5402e8fb96aabbf368d78" candidate_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999e3a35b4d4491b9b1cd70054206eec" is_pre_election: true
I20260924 07:22:15.540347 22827 raft_consensus.cc:2454] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99d41d1a8bcd4c4bb0cd2ea8d1a31756 in term 0.
I20260924 07:22:15.540735 23157 raft_consensus.cc:504] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999e3a35b4d4491b9b1cd70054206eec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34343 } } peers { permanent_uuid: "a8b3ff4484044a7d8eb4ec1983eeed99" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40707 } } peers { permanent_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 39135 } }
I20260924 07:22:15.540869 23157 leader_election.cc:288] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [CANDIDATE]: Term 1 election: Requested vote from peers 999e3a35b4d4491b9b1cd70054206eec (127.3.246.65:34343), a8b3ff4484044a7d8eb4ec1983eeed99 (127.3.246.67:40707)
I20260924 07:22:15.540990 22827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f881088faaa5402e8fb96aabbf368d78" candidate_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999e3a35b4d4491b9b1cd70054206eec"
I20260924 07:22:15.541004 23089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f881088faaa5402e8fb96aabbf368d78" candidate_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8b3ff4484044a7d8eb4ec1983eeed99"
I20260924 07:22:15.541085 23089 raft_consensus.cc:3037] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:15.541087 22827 raft_consensus.cc:3037] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:15.541869 23089 raft_consensus.cc:2454] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d41d1a8bcd4c4bb0cd2ea8d1a31756 in term 1.
I20260924 07:22:15.541949 22827 raft_consensus.cc:2454] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d41d1a8bcd4c4bb0cd2ea8d1a31756 in term 1.
I20260924 07:22:15.542025 22894 leader_election.cc:302] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99d41d1a8bcd4c4bb0cd2ea8d1a31756, a8b3ff4484044a7d8eb4ec1983eeed99; no voters:
I20260924 07:22:15.542119 23157 raft_consensus.cc:2781] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:15.542258 23157 raft_consensus.cc:686] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [term 1 LEADER]: Becoming Leader. State: Replica: 99d41d1a8bcd4c4bb0cd2ea8d1a31756, State: Running, Role: LEADER
I20260924 07:22:15.542363 23157 consensus_queue.cc:237] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999e3a35b4d4491b9b1cd70054206eec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34343 } } peers { permanent_uuid: "a8b3ff4484044a7d8eb4ec1983eeed99" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40707 } } peers { permanent_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 39135 } }
I20260924 07:22:15.543148 22684 catalog_manager.cc:5495] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99d41d1a8bcd4c4bb0cd2ea8d1a31756 (127.3.246.66). New cstate: current_term: 1 leader_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999e3a35b4d4491b9b1cd70054206eec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8b3ff4484044a7d8eb4ec1983eeed99" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d41d1a8bcd4c4bb0cd2ea8d1a31756" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 39135 } health_report { overall_health: HEALTHY } } }
I20260924 07:22:15.568634 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.566362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.3.246.66:39135)
0924 07:22:15.566425 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:15.566427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:15.566438 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:22:15.566497 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:15.566498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:15.566515 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:15.566571 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:15.566691 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.567164 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.567852 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:15.568042 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:15.568120 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:15.568240 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:15.568325 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:15.568393 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:15.568490 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 07:22:15.568520 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:22:15.570464 22827 raft_consensus.cc:1261] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec [term 1 FOLLOWER]: Refusing update from remote peer 99d41d1a8bcd4c4bb0cd2ea8d1a31756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:15.570464 23089 raft_consensus.cc:1261] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99 [term 1 FOLLOWER]: Refusing update from remote peer 99d41d1a8bcd4c4bb0cd2ea8d1a31756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:15.570820 23157 consensus_queue.cc:1035] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8b3ff4484044a7d8eb4ec1983eeed99" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:15.570974 23159 consensus_queue.cc:1035] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "999e3a35b4d4491b9b1cd70054206eec" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 34343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:15.573220 23162 mvcc.cc:205] Tried to move back new op lower bound from 7332800657693597696 to 7332800657582178304. Current Snapshot: MvccSnapshot[applied={T|T < 7332800657693597696}]
I20260924 07:22:15.575776 22685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:22:15.575868 22685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:22:15.576615 22685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca831585d7014381adf52dfffe54f68e: Sending DeleteTablet for 3 replicas of tablet f881088faaa5402e8fb96aabbf368d78
I20260924 07:22:15.576949 22938 tablet_service.cc:1514] Processing DeleteTablet for tablet f881088faaa5402e8fb96aabbf368d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:15 UTC) from {username='slave'} at 127.0.0.1:42740
I20260924 07:22:15.577010 22807 tablet_service.cc:1514] Processing DeleteTablet for tablet f881088faaa5402e8fb96aabbf368d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:15 UTC) from {username='slave'} at 127.0.0.1:58882
I20260924 07:22:15.577215 23172 tablet_replica.cc:330] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756: stopping tablet replica
I20260924 07:22:15.577303 23173 tablet_replica.cc:330] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec: stopping tablet replica
I20260924 07:22:15.577273 23069 tablet_service.cc:1514] Processing DeleteTablet for tablet f881088faaa5402e8fb96aabbf368d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:15 UTC) from {username='slave'} at 127.0.0.1:43342
I20260924 07:22:15.577311 23172 raft_consensus.cc:2229] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:22:15.577418 23172 raft_consensus.cc:2258] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:15.577500 23173 raft_consensus.cc:2229] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:22:15.577663 23173 raft_consensus.cc:2258] T f881088faaa5402e8fb96aabbf368d78 P 999e3a35b4d4491b9b1cd70054206eec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:15.577733 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 22745
I20260924 07:22:15.578619 23172 ts_tablet_manager.cc:1905] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:22:15.580307 23172 ts_tablet_manager.cc:1918] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:22:15.580389 23172 log.cc:1196] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/f881088faaa5402e8fb96aabbf368d78
I20260924 07:22:15.580632 23172 ts_tablet_manager.cc:1939] T f881088faaa5402e8fb96aabbf368d78 P 99d41d1a8bcd4c4bb0cd2ea8d1a31756: Deleting consensus metadata
I20260924 07:22:15.581012 22669 catalog_manager.cc:4841] TS 99d41d1a8bcd4c4bb0cd2ea8d1a31756 (127.3.246.66:39135): tablet f881088faaa5402e8fb96aabbf368d78 (table test-table [id=a3d878c84d2c40388bd7305c4db3a602]) successfully deleted
W20260924 07:22:15.583595 22671 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:34343 (error 108)
I20260924 07:22:15.583854 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 22876
I20260924 07:22:15.584120 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:15.584019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:34343 (local address 127.0.0.1:58888)
0924 07:22:15.584061 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:15.584091 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:34343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 07:22:15.584224 22671 catalog_manager.cc:4584] TS 999e3a35b4d4491b9b1cd70054206eec (127.3.246.65:34343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f881088faaa5402e8fb96aabbf368d78: Network error: Client connection negotiation failed: client connection to 127.3.246.65:34343: connect: Connection refused (error 111)
I20260924 07:22:15.584511 23174 tablet_replica.cc:330] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99: stopping tablet replica
I20260924 07:22:15.584653 23174 raft_consensus.cc:2229] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:22:15.584764 23174 raft_consensus.cc:2258] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:15.585130 23174 ts_tablet_manager.cc:1905] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:22:15.586654 23174 ts_tablet_manager.cc:1918] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:22:15.586738 23174 log.cc:1196] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/f881088faaa5402e8fb96aabbf368d78
I20260924 07:22:15.586973 23174 ts_tablet_manager.cc:1939] T f881088faaa5402e8fb96aabbf368d78 P a8b3ff4484044a7d8eb4ec1983eeed99: Deleting consensus metadata
I20260924 07:22:15.587356 22672 catalog_manager.cc:4841] TS a8b3ff4484044a7d8eb4ec1983eeed99 (127.3.246.67:40707): tablet f881088faaa5402e8fb96aabbf368d78 (table test-table [id=a3d878c84d2c40388bd7305c4db3a602]) successfully deleted
I20260924 07:22:15.589627 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 23007
I20260924 07:22:15.594784 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 22654
2026-09-24T07:22:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (792 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:15 dist-test-slave-rc1w krb5kdc[23178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:15 dist-test-slave-rc1w krb5kdc[23178](info): set up 2 sockets
Sep 24 07:22:15 dist-test-slave-rc1w krb5kdc[23178](info): commencing operation
krb5kdc: starting...
W20260924 07:22:16.629540 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:16 dist-test-slave-rc1w krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:22:16.745203 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37421
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:37421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:16.824903 23191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:16.825098 23191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:16.825120 23191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:16.826202 23191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:16.826234 23191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:16.826248 23191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:16.826260 23191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:16.826272 23191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:16.827625 23191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:37421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:16.827797 23191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:16.827955 23191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:16.832902 23197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:16.832918 23196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:16.832952 23191 server_base.cc:1034] running on GCE node
W20260924 07:22:16.832918 23199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:16.833359 23191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:16.833644 23191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:16.834829 23191 hybrid_clock.cc:648] HybridClock initialized: now 1790234536834816 us; error 29 us; skew 500 ppm
Sep 24 07:22:16 dist-test-slave-rc1w krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234536, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:16.837225 23191 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:22:16.837575 23191 webserver.cc:460] Webserver started at http://127.3.246.126:44861/ using document root <none> and password file <none>
I20260924 07:22:16.837831 23191 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:16.837878 23191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:16.838025 23191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:16.839030 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "a015e1d7bb0848599d2d232ef4e77d9f"
format_stamp: "Formatted at 2026-09-24 07:22:16 on dist-test-slave-rc1w"
server_key: "82a45892990cff6371d3f65fb4530da0"
server_key_iv: "78da9b7fe41530a41b22979c5d946095"
server_key_version: "encryptionkey@0"
I20260924 07:22:16.839381 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "a015e1d7bb0848599d2d232ef4e77d9f"
format_stamp: "Formatted at 2026-09-24 07:22:16 on dist-test-slave-rc1w"
server_key: "82a45892990cff6371d3f65fb4530da0"
server_key_iv: "78da9b7fe41530a41b22979c5d946095"
server_key_version: "encryptionkey@0"
I20260924 07:22:16.840658 23191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:22:16.841394 23206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:16.841645 23191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:22:16.841787 23191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "a015e1d7bb0848599d2d232ef4e77d9f"
format_stamp: "Formatted at 2026-09-24 07:22:16 on dist-test-slave-rc1w"
server_key: "82a45892990cff6371d3f65fb4530da0"
server_key_iv: "78da9b7fe41530a41b22979c5d946095"
server_key_version: "encryptionkey@0"
I20260924 07:22:16.841862 23191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:16.869869 23191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:16.871806 23191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:16.871948 23191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:16.875792 23191 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:37421
I20260924 07:22:16.875847 23258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:37421 every 8 connection(s)
I20260924 07:22:16.876206 23191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:16.877053 23259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:16.879307 23259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f: Bootstrap starting.
I20260924 07:22:16.879875 23259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:16.880205 23259 log.cc:824] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:16.880244 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 23191
I20260924 07:22:16.880344 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:16.880554 4057 external_mini_cluster.cc:1428] Setting key a88e72b8b326d5495bf9dc759e79278a
I20260924 07:22:16.881088 23259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f: No bootstrap required, opened a new log
I20260924 07:22:16.882817 23259 raft_consensus.cc:348] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a015e1d7bb0848599d2d232ef4e77d9f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37421 } }
I20260924 07:22:16.882973 23259 raft_consensus.cc:374] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:16.883019 23259 raft_consensus.cc:729] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a015e1d7bb0848599d2d232ef4e77d9f, State: Initialized, Role: FOLLOWER
I20260924 07:22:16.883179 23259 consensus_queue.cc:260] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a015e1d7bb0848599d2d232ef4e77d9f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37421 } }
I20260924 07:22:16.883267 23259 raft_consensus.cc:388] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:16.883319 23259 raft_consensus.cc:482] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:16.883371 23259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:16.884191 23259 raft_consensus.cc:504] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a015e1d7bb0848599d2d232ef4e77d9f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37421 } }
I20260924 07:22:16.884318 23259 leader_election.cc:302] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a015e1d7bb0848599d2d232ef4e77d9f; no voters:
I20260924 07:22:16.884493 23259 leader_election.cc:288] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:16.884550 23264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:16.884680 23264 raft_consensus.cc:686] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [term 1 LEADER]: Becoming Leader. State: Replica: a015e1d7bb0848599d2d232ef4e77d9f, State: Running, Role: LEADER
I20260924 07:22:16.884755 23259 sys_catalog.cc:564] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:16.884773 23264 consensus_queue.cc:237] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a015e1d7bb0848599d2d232ef4e77d9f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37421 } }
I20260924 07:22:16.885115 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a015e1d7bb0848599d2d232ef4e77d9f. Latest consensus state: current_term: 1 leader_uuid: "a015e1d7bb0848599d2d232ef4e77d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a015e1d7bb0848599d2d232ef4e77d9f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37421 } } }
I20260924 07:22:16.885250 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:16.885131 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a015e1d7bb0848599d2d232ef4e77d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a015e1d7bb0848599d2d232ef4e77d9f" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37421 } } }
I20260924 07:22:16.885418 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:16.885439 23271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:16.886083 23271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:16.888942 23271 catalog_manager.cc:1347] Generated new cluster ID: 789ca9f6dbc54d939ceced6dace76698
I20260924 07:22:16.888996 23271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:22:16 dist-test-slave-rc1w krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:16.892870 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:16.881080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59760 (local address 127.3.246.126:37421)
0924 07:22:16.881280 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:16.881289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:16.881324 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:22:16.882814 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:16.882822 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:16.882874 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:16.883077 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:16.883839 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:16.885092 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:16.886335 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:16.886863 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:16.887003 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:16.889609 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:16.889613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:16.889613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:16.889626 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:16.891362 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:16.891739 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:16.891742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:16.891742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:16.891851 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:16.892061 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:16.892062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:16.892062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:16.892280 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:16.892436 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:16.892555 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 07:22:16.892693 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260924 07:22:16.919477 23271 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:16.920022 23271 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:16.925408 23271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f: Generated new TSK 0
I20260924 07:22:16.925608 23271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
Entry for principal kudu/127.3.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab.
I20260924 07:22:16.965008 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37421
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:17.058547 23285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:17.058761 23285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:17.058787 23285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:17.059861 23285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:17.059899 23285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:17.059926 23285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:22:17.061273 23285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.65.keytab
--principal=kudu/127.3.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:17.061462 23285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:17.061641 23285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:17.066731 23290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:17.066725 23293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:17.066725 23291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:17.066941 23285 server_base.cc:1034] running on GCE node
I20260924 07:22:17.067233 23285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:17.067471 23285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:17.068640 23285 hybrid_clock.cc:648] HybridClock initialized: now 1790234537068630 us; error 39 us; skew 500 ppm
Sep 24 07:22:17 dist-test-slave-rc1w krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234537, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:17.071171 23285 init.cc:377] Logged in from keytab as kudu/127.3.246.65@KRBTEST.COM (short username kudu)
I20260924 07:22:17.071624 23285 webserver.cc:460] Webserver started at http://127.3.246.65:38213/ using document root <none> and password file <none>
I20260924 07:22:17.071837 23285 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:17.071904 23285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:17.072057 23285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:17.073081 23285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "2adf6ed92b7945ee812b2f165f3e3662"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "0f65d39a82c899c8aca6640cbf2653c3"
server_key_iv: "1ff588c45d20adba34bd3bf79c71ec34"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.073437 23285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "2adf6ed92b7945ee812b2f165f3e3662"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "0f65d39a82c899c8aca6640cbf2653c3"
server_key_iv: "1ff588c45d20adba34bd3bf79c71ec34"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.074805 23285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 07:22:17.075544 23300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.075722 23285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:17.075825 23285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "2adf6ed92b7945ee812b2f165f3e3662"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "0f65d39a82c899c8aca6640cbf2653c3"
server_key_iv: "1ff588c45d20adba34bd3bf79c71ec34"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.075911 23285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:17.098970 23285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:17.101400 23285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:17.101579 23285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:17.101878 23285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:17.102198 23285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:17.102253 23285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.102295 23285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:17.102342 23285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.108109 23285 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:37627
I20260924 07:22:17.108160 23413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:37627 every 8 connection(s)
I20260924 07:22:17.108467 23285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:22:17.110384 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 23285
I20260924 07:22:17.110527 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:22:17.110795 4057 external_mini_cluster.cc:1428] Setting key 254ff9b0a8e2b3e2868c4e26950c79e9
Sep 24 07:22:17 dist-test-slave-rc1w krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234537, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.65@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.246.66@KRBTEST.COM; defaulting to no policy
I20260924 07:22:17.119342 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.109177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:48629 (local address 127.3.246.126:37421)
0924 07:22:17.109268 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:17.109270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:17.109434 (+ 164us) server_negotiation.cc:407] Connection header received
0924 07:22:17.109952 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:17.109953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:17.109968 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:17.110033 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:17.111372 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.112198 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.113238 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.113588 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.113691 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.116095 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:17.116097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:17.116097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:17.116103 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:17.117618 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:17.118190 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:17.118192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:17.118192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:17.118248 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:17.118592 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:17.118594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:17.118594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:17.118721 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:17.118800 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:17.119129 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 07:22:17.119273 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:22:17.119551 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.109114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37421 (local address 127.3.246.65:48629)
0924 07:22:17.109312 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:17.109349 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:17.109856 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:17.110087 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:17.110090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:17.110941 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:17.111253 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.111261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.112299 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.112300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.113148 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.113151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.113728 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.113729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.113869 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.114415 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:17.114423 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:17.115978 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:17.117769 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:17.117777 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:17.117779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:17.118084 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:17.118330 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:17.118331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:17.118331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:17.118474 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:17.118814 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:17.118822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:17.118981 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 07:22:17.119111 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 07:22:17.119291 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260924 07:22:17.120026 23414 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37421
I20260924 07:22:17.120143 23414 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:17.120328 23414 heartbeater.cc:511] Master 127.3.246.126:37421 requested a full tablet report, sending...
I20260924 07:22:17.120890 23223 ts_manager.cc:194] Registered new tserver with Master: 2adf6ed92b7945ee812b2f165f3e3662 (127.3.246.65:37627)
I20260924 07:22:17.121598 23223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.65@KRBTEST.COM'} at 127.3.246.65:48629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
Entry for principal kudu/127.3.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab.
I20260924 07:22:17.134665 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37421
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:17.216849 23419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:17.217053 23419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:17.217077 23419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:17.218164 23419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:17.218199 23419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:17.218225 23419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:22:17.219631 23419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.66.keytab
--principal=kudu/127.3.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:17.219800 23419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:17.219960 23419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:17.224830 23425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:17.224843 23427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:17.224843 23424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:17.224948 23419 server_base.cc:1034] running on GCE node
I20260924 07:22:17.225266 23419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:17.225489 23419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:17.226661 23419 hybrid_clock.cc:648] HybridClock initialized: now 1790234537226645 us; error 36 us; skew 500 ppm
Sep 24 07:22:17 dist-test-slave-rc1w krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234537, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:17.229068 23419 init.cc:377] Logged in from keytab as kudu/127.3.246.66@KRBTEST.COM (short username kudu)
I20260924 07:22:17.229518 23419 webserver.cc:460] Webserver started at http://127.3.246.66:39021/ using document root <none> and password file <none>
I20260924 07:22:17.229789 23419 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:17.229836 23419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:17.229970 23419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:17.231002 23419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "cce187e9eb654c958d528ee68e7f124f"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "815b68f3fecd4e4b5773d63077b325d7"
server_key_iv: "97e2181060b465dd8e05b0366c903166"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.231356 23419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "cce187e9eb654c958d528ee68e7f124f"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "815b68f3fecd4e4b5773d63077b325d7"
server_key_iv: "97e2181060b465dd8e05b0366c903166"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.232728 23419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:22:17.233486 23434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.233737 23419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:22:17.233830 23419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "cce187e9eb654c958d528ee68e7f124f"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "815b68f3fecd4e4b5773d63077b325d7"
server_key_iv: "97e2181060b465dd8e05b0366c903166"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.233904 23419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:17.254352 23419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:17.256218 23419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:17.256385 23419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:17.256618 23419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:17.256953 23419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:17.256986 23419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.257009 23419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:17.257053 23419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.262837 23419 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:33691
I20260924 07:22:17.262887 23547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:33691 every 8 connection(s)
I20260924 07:22:17.263239 23419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:22:17.269119 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 23419
I20260924 07:22:17.269248 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:22:17.269450 4057 external_mini_cluster.cc:1428] Setting key ab7142d9d4e764617d59fc1a5d990ffd
Sep 24 07:22:17 dist-test-slave-rc1w krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234537, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.66@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:17.272795 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.263921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:41915 (local address 127.3.246.126:37421)
0924 07:22:17.263986 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:17.263988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:17.264186 (+ 198us) server_negotiation.cc:407] Connection header received
0924 07:22:17.264678 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:17.264679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:17.264689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:17.264756 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:17.265907 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.266404 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.267373 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.267597 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.267650 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.270132 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:17.270133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:17.270133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:17.270141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:17.271423 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:17.271819 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:17.271820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:17.271821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:17.271885 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:17.272147 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:17.272148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:17.272148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:17.272266 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:17.272339 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:17.272611 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 07:22:17.272707 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:22:17.272971 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.263874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37421 (local address 127.3.246.66:41915)
0924 07:22:17.264063 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:17.264104 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:17.264578 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:17.264933 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:17.264937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:17.265534 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:17.265822 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.265827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.266487 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.266487 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.267294 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.267296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.267675 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.267676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.267776 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.268337 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:17.268345 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:17.270049 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:17.271509 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:17.271516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:17.271518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:17.271740 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:17.271960 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:17.271961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:17.271961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:17.272076 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:17.272343 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:17.272350 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:17.272482 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 07:22:17.272591 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 07:22:17.272743 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":125,"threads_started":1}
I20260924 07:22:17.273495 23548 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37421
I20260924 07:22:17.273593 23548 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:17.273809 23548 heartbeater.cc:511] Master 127.3.246.126:37421 requested a full tablet report, sending...
I20260924 07:22:17.274206 23223 ts_manager.cc:194] Registered new tserver with Master: cce187e9eb654c958d528ee68e7f124f (127.3.246.66:33691)
I20260924 07:22:17.274763 23223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.66@KRBTEST.COM'} at 127.3.246.66:41915
WARNING: no policy specified for kudu/127.3.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
Entry for principal kudu/127.3.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab.
I20260924 07:22:17.292850 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37421
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:17.371416 23553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:17.371604 23553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:17.371627 23553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:17.372768 23553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:17.372831 23553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:17.372875 23553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:22:17.374307 23553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.67.keytab
--principal=kudu/127.3.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:37421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:17.374568 23553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:17.374783 23553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:17.379515 23558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:17.379554 23559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:17.379535 23561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:17.379892 23553 server_base.cc:1034] running on GCE node
I20260924 07:22:17.380096 23553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:17.380295 23553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:17.381429 23553 hybrid_clock.cc:648] HybridClock initialized: now 1790234537381414 us; error 31 us; skew 500 ppm
Sep 24 07:22:17 dist-test-slave-rc1w krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234537, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:17.383914 23553 init.cc:377] Logged in from keytab as kudu/127.3.246.67@KRBTEST.COM (short username kudu)
I20260924 07:22:17.384439 23553 webserver.cc:460] Webserver started at http://127.3.246.67:38433/ using document root <none> and password file <none>
I20260924 07:22:17.384697 23553 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:17.384768 23553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:17.384922 23553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:17.386188 23553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "f9a389e1db9d43bf8aa4a7b40279131e"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "c91d11d8aa5a78496f272db787b6baed"
server_key_iv: "6bfe2b35c55f40d24b1375756a257817"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.386647 23553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "f9a389e1db9d43bf8aa4a7b40279131e"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "c91d11d8aa5a78496f272db787b6baed"
server_key_iv: "6bfe2b35c55f40d24b1375756a257817"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.388038 23553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 07:22:17.388834 23568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.389017 23553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:22:17.389114 23553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "f9a389e1db9d43bf8aa4a7b40279131e"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "c91d11d8aa5a78496f272db787b6baed"
server_key_iv: "6bfe2b35c55f40d24b1375756a257817"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.389207 23553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:17.398581 23553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:17.400410 23553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:17.400568 23553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:17.400784 23553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:17.401079 23553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:17.401134 23553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.401175 23553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:17.401203 23553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.406975 23553 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:38137
I20260924 07:22:17.407048 23681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:38137 every 8 connection(s)
I20260924 07:22:17.407377 23553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:22:17 dist-test-slave-rc1w krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234537, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.67@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:17.416319 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.408117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:55187 (local address 127.3.246.126:37421)
0924 07:22:17.408180 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:17.408182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:17.408392 (+ 210us) server_negotiation.cc:407] Connection header received
0924 07:22:17.408915 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:17.408916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:17.408928 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:17.408996 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:17.409992 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.410500 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.411498 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.411709 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.411788 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.413769 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:17.413770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:17.413770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:17.413778 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:17.414932 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:17.415347 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:17.415348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:17.415349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:17.415408 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:17.415665 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:17.415666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:17.415666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:17.415794 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:17.415871 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:17.416147 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 07:22:17.416229 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:22:17.416495 23684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.408081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:37421 (local address 127.3.246.67:55187)
0924 07:22:17.408269 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:17.408311 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:17.408809 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:17.409064 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:17.409068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:17.409631 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:17.409896 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.409902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.410611 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.410612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.411394 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.411397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.411794 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.411796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.411923 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.412348 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:17.412355 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:17.413619 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:17.415033 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:17.415040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:17.415042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:17.415259 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:17.415477 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:17.415477 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:17.415478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:17.415593 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:17.415875 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:17.415882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:17.416013 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 07:22:17.416112 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 07:22:17.416269 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20260924 07:22:17.417047 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 23553
I20260924 07:22:17.417042 23682 heartbeater.cc:347] Connected to a master server at 127.3.246.126:37421
I20260924 07:22:17.417158 23682 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:17.417160 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:22:17.417330 23682 heartbeater.cc:511] Master 127.3.246.126:37421 requested a full tablet report, sending...
I20260924 07:22:17.417336 4057 external_mini_cluster.cc:1428] Setting key e3373bf280705263450d079dad9c90c7
I20260924 07:22:17.417754 23223 ts_manager.cc:194] Registered new tserver with Master: f9a389e1db9d43bf8aa4a7b40279131e (127.3.246.67:38137)
I20260924 07:22:17.418236 23223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.246.67@KRBTEST.COM'} at 127.3.246.67:55187
I20260924 07:22:17.418311 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:22:17.425064 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.419581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59776 (local address 127.3.246.126:37421)
0924 07:22:17.419625 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:17.419627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:17.419780 (+ 153us) server_negotiation.cc:407] Connection header received
0924 07:22:17.419817 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:17.419818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:17.419826 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:17.419862 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:17.420394 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.420826 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.421538 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.421780 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.421834 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.423041 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:17.423042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:17.423042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:17.423047 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:17.424163 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:17.424477 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:17.424478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:17.424478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:17.424523 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:17.424659 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:17.424659 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:17.424659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:17.424779 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:17.424851 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:17.424956 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 07:22:17.425009 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:22:17.426964 23223 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:22:17.428512 23223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:22:17.435442 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.430878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:38137 (local address 127.0.0.1:45058)
0924 07:22:17.431053 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:17.431095 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:17.431161 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:17.431885 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:17.431886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:17.431888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:17.431949 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.431951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.433148 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.433149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.434154 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.434157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.435174 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.435175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.435246 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.435250 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:17.435298 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 07:22:17.435357 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":113,"threads_started":1}
I20260924 07:22:17.435526 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.430999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.3.246.66:33691)
0924 07:22:17.431827 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:17.431831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:17.431841 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:22:17.431883 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:17.431884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:17.431899 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:17.432135 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:17.432195 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.432835 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.434464 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.435174 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.435246 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.435300 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:17.435399 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 07:22:17.435438 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":73,"threads_started":1}
I20260924 07:22:17.435768 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.430970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.3.246.67:38137)
0924 07:22:17.431538 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:17.431543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:17.431554 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:22:17.431592 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:17.431594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:17.431610 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:17.431687 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:17.432031 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.433044 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.434264 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.435006 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.435081 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.435136 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:17.435652 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 07:22:17.435716 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":72,"threads_started":1}
I20260924 07:22:17.435527 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.430881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:33691 (local address 127.0.0.1:52574)
0924 07:22:17.431233 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:17.431242 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:17.431286 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:17.432026 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:17.432027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:17.432028 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:17.432059 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.432060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.432958 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.432960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.434369 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.434372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.435262 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.435263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.435318 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.435321 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:22:17.435384 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 07:22:17.435422 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":46,"thread_start_us":50,"threads_started":1}
I20260924 07:22:17.436117 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.431344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.3.246.65:37627)
0924 07:22:17.431542 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:17.431547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:17.431564 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:22:17.431617 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:17.431619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:17.431635 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:17.431731 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:17.431869 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.433005 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.434485 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.434982 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.435090 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.435151 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:17.435410 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 07:22:17.435469 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":97,"threads_started":1}
I20260924 07:22:17.436075 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.431105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:37627 (local address 127.0.0.1:53342)
0924 07:22:17.431344 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:17.431355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:17.431406 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:17.431741 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:17.431745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:17.431747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:17.431808 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.431810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.433112 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.433113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.434369 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.434372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.435094 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.435095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.435186 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.435200 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 07:22:17.435272 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:22:17.435333 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":34,"thread_start_us":91,"threads_started":1}
I20260924 07:22:17.436923 23482 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9f2ff7879f41aa80a46ea0d48e40b5 (DEFAULT_TABLE table=test-table [id=87624f12f1ac478d81d34d86dbb11ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:17.436988 23616 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9f2ff7879f41aa80a46ea0d48e40b5 (DEFAULT_TABLE table=test-table [id=87624f12f1ac478d81d34d86dbb11ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:17.437000 23348 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9f2ff7879f41aa80a46ea0d48e40b5 (DEFAULT_TABLE table=test-table [id=87624f12f1ac478d81d34d86dbb11ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:17.437295 23616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9f2ff7879f41aa80a46ea0d48e40b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:17.437295 23482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9f2ff7879f41aa80a46ea0d48e40b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:17.437311 23348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9f2ff7879f41aa80a46ea0d48e40b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:17.440141 23699 tablet_bootstrap.cc:492] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662: Bootstrap starting.
I20260924 07:22:17.440878 23699 tablet_bootstrap.cc:654] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:17.440871 23700 tablet_bootstrap.cc:492] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e: Bootstrap starting.
I20260924 07:22:17.441480 23699 log.cc:824] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:17.441535 23700 tablet_bootstrap.cc:654] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:17.441828 23701 tablet_bootstrap.cc:492] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f: Bootstrap starting.
I20260924 07:22:17.442157 23700 log.cc:824] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:17.442365 23699 tablet_bootstrap.cc:492] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662: No bootstrap required, opened a new log
I20260924 07:22:17.442442 23699 ts_tablet_manager.cc:1397] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:22:17.442646 23701 tablet_bootstrap.cc:654] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:17.442886 23700 tablet_bootstrap.cc:492] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e: No bootstrap required, opened a new log
I20260924 07:22:17.442952 23700 ts_tablet_manager.cc:1397] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:22:17.443222 23701 log.cc:824] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:17.444002 23701 tablet_bootstrap.cc:492] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f: No bootstrap required, opened a new log
I20260924 07:22:17.444057 23701 ts_tablet_manager.cc:1397] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:22:17.444026 23699 raft_consensus.cc:348] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce187e9eb654c958d528ee68e7f124f" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33691 } } peers { permanent_uuid: "f9a389e1db9d43bf8aa4a7b40279131e" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38137 } } peers { permanent_uuid: "2adf6ed92b7945ee812b2f165f3e3662" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 37627 } }
I20260924 07:22:17.444152 23699 raft_consensus.cc:374] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:17.444175 23699 raft_consensus.cc:729] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2adf6ed92b7945ee812b2f165f3e3662, State: Initialized, Role: FOLLOWER
I20260924 07:22:17.444298 23699 consensus_queue.cc:260] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce187e9eb654c958d528ee68e7f124f" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33691 } } peers { permanent_uuid: "f9a389e1db9d43bf8aa4a7b40279131e" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38137 } } peers { permanent_uuid: "2adf6ed92b7945ee812b2f165f3e3662" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 37627 } }
I20260924 07:22:17.444579 23414 heartbeater.cc:503] Master 127.3.246.126:37421 was elected leader, sending a full tablet report...
I20260924 07:22:17.444579 23699 ts_tablet_manager.cc:1428] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:22:17.444971 23700 raft_consensus.cc:348] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce187e9eb654c958d528ee68e7f124f" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33691 } } peers { permanent_uuid: "f9a389e1db9d43bf8aa4a7b40279131e" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38137 } } peers { permanent_uuid: "2adf6ed92b7945ee812b2f165f3e3662" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 37627 } }
I20260924 07:22:17.445122 23700 raft_consensus.cc:374] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:17.445155 23700 raft_consensus.cc:729] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a389e1db9d43bf8aa4a7b40279131e, State: Initialized, Role: FOLLOWER
I20260924 07:22:17.445263 23700 consensus_queue.cc:260] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce187e9eb654c958d528ee68e7f124f" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33691 } } peers { permanent_uuid: "f9a389e1db9d43bf8aa4a7b40279131e" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38137 } } peers { permanent_uuid: "2adf6ed92b7945ee812b2f165f3e3662" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 37627 } }
I20260924 07:22:17.445525 23700 ts_tablet_manager.cc:1428] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:22:17.445655 23682 heartbeater.cc:503] Master 127.3.246.126:37421 was elected leader, sending a full tablet report...
I20260924 07:22:17.446091 23701 raft_consensus.cc:348] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce187e9eb654c958d528ee68e7f124f" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33691 } } peers { permanent_uuid: "f9a389e1db9d43bf8aa4a7b40279131e" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38137 } } peers { permanent_uuid: "2adf6ed92b7945ee812b2f165f3e3662" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 37627 } }
I20260924 07:22:17.446245 23701 raft_consensus.cc:374] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:17.446285 23701 raft_consensus.cc:729] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce187e9eb654c958d528ee68e7f124f, State: Initialized, Role: FOLLOWER
I20260924 07:22:17.446394 23701 consensus_queue.cc:260] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce187e9eb654c958d528ee68e7f124f" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33691 } } peers { permanent_uuid: "f9a389e1db9d43bf8aa4a7b40279131e" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38137 } } peers { permanent_uuid: "2adf6ed92b7945ee812b2f165f3e3662" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 37627 } }
I20260924 07:22:17.446656 23701 ts_tablet_manager.cc:1428] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:22:17.446697 23548 heartbeater.cc:503] Master 127.3.246.126:37421 was elected leader, sending a full tablet report...
I20260924 07:22:17.478742 23707 raft_consensus.cc:482] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:22:17.478869 23707 raft_consensus.cc:504] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce187e9eb654c958d528ee68e7f124f" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33691 } } peers { permanent_uuid: "f9a389e1db9d43bf8aa4a7b40279131e" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38137 } } peers { permanent_uuid: "2adf6ed92b7945ee812b2f165f3e3662" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 37627 } }
I20260924 07:22:17.479213 23707 leader_election.cc:288] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9a389e1db9d43bf8aa4a7b40279131e (127.3.246.67:38137), 2adf6ed92b7945ee812b2f165f3e3662 (127.3.246.65:37627)
I20260924 07:22:17.482079 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.479508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:53599 (local address 127.3.246.67:38137)
0924 07:22:17.479571 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:17.479578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:17.479586 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:22:17.479613 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:17.479614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:17.479622 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:17.479707 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:17.479873 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.480348 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.481278 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.481654 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.481803 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.481813 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:17.481979 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 07:22:17.482020 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:22:17.482074 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.479439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:38137 (local address 127.3.246.66:53599)
0924 07:22:17.479497 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:17.479510 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:17.479557 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:17.479713 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:17.479715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:17.479717 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:17.479794 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.479796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.480433 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.480434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.481170 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.481172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.481804 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.481805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.481893 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.481896 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:22:17.481960 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:22:17.482001 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 07:22:17.482402 23636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9f2ff7879f41aa80a46ea0d48e40b5" candidate_uuid: "cce187e9eb654c958d528ee68e7f124f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a389e1db9d43bf8aa4a7b40279131e" is_pre_election: true
I20260924 07:22:17.482542 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.479704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:34317 (local address 127.3.246.65:37627)
0924 07:22:17.479753 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:17.479755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:17.479831 (+ 76us) server_negotiation.cc:407] Connection header received
0924 07:22:17.479915 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:17.479916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:17.479926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:17.480024 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:17.480204 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.480681 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.481639 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.482098 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.482200 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.482210 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:17.482435 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 07:22:17.482480 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:22:17.482549 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.479567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:37627 (local address 127.3.246.66:34317)
0924 07:22:17.479755 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:17.479764 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:17.479845 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:17.480046 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:17.480047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:17.480049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:17.480127 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.480129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.480776 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.480778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.481538 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.481541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.482255 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.482257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.482345 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.482349 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:17.482428 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:22:17.482467 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":112,"threads_started":1}
I20260924 07:22:17.482545 23636 raft_consensus.cc:2454] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cce187e9eb654c958d528ee68e7f124f in term 0.
I20260924 07:22:17.482856 23438 leader_election.cc:302] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cce187e9eb654c958d528ee68e7f124f, f9a389e1db9d43bf8aa4a7b40279131e; no voters:
I20260924 07:22:17.482983 23707 raft_consensus.cc:2781] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:22:17.483042 23707 raft_consensus.cc:482] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:22:17.483072 23707 raft_consensus.cc:3037] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:17.483008 23368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9f2ff7879f41aa80a46ea0d48e40b5" candidate_uuid: "cce187e9eb654c958d528ee68e7f124f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2adf6ed92b7945ee812b2f165f3e3662" is_pre_election: true
I20260924 07:22:17.483173 23368 raft_consensus.cc:2454] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cce187e9eb654c958d528ee68e7f124f in term 0.
I20260924 07:22:17.483925 23707 raft_consensus.cc:504] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce187e9eb654c958d528ee68e7f124f" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33691 } } peers { permanent_uuid: "f9a389e1db9d43bf8aa4a7b40279131e" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38137 } } peers { permanent_uuid: "2adf6ed92b7945ee812b2f165f3e3662" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 37627 } }
I20260924 07:22:17.484076 23707 leader_election.cc:288] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [CANDIDATE]: Term 1 election: Requested vote from peers f9a389e1db9d43bf8aa4a7b40279131e (127.3.246.67:38137), 2adf6ed92b7945ee812b2f165f3e3662 (127.3.246.65:37627)
I20260924 07:22:17.484257 23636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9f2ff7879f41aa80a46ea0d48e40b5" candidate_uuid: "cce187e9eb654c958d528ee68e7f124f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a389e1db9d43bf8aa4a7b40279131e"
I20260924 07:22:17.484345 23636 raft_consensus.cc:3037] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:17.485132 23636 raft_consensus.cc:2454] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cce187e9eb654c958d528ee68e7f124f in term 1.
I20260924 07:22:17.485225 23368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9f2ff7879f41aa80a46ea0d48e40b5" candidate_uuid: "cce187e9eb654c958d528ee68e7f124f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2adf6ed92b7945ee812b2f165f3e3662"
I20260924 07:22:17.485309 23438 leader_election.cc:302] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cce187e9eb654c958d528ee68e7f124f, f9a389e1db9d43bf8aa4a7b40279131e; no voters:
I20260924 07:22:17.485324 23368 raft_consensus.cc:3037] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:17.485416 23707 raft_consensus.cc:2781] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:17.485615 23707 raft_consensus.cc:686] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [term 1 LEADER]: Becoming Leader. State: Replica: cce187e9eb654c958d528ee68e7f124f, State: Running, Role: LEADER
I20260924 07:22:17.485734 23707 consensus_queue.cc:237] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce187e9eb654c958d528ee68e7f124f" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33691 } } peers { permanent_uuid: "f9a389e1db9d43bf8aa4a7b40279131e" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38137 } } peers { permanent_uuid: "2adf6ed92b7945ee812b2f165f3e3662" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 37627 } }
I20260924 07:22:17.486219 23368 raft_consensus.cc:2454] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cce187e9eb654c958d528ee68e7f124f in term 1.
I20260924 07:22:17.486621 23222 catalog_manager.cc:5495] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f reported cstate change: term changed from 0 to 1, leader changed from <none> to cce187e9eb654c958d528ee68e7f124f (127.3.246.66). New cstate: current_term: 1 leader_uuid: "cce187e9eb654c958d528ee68e7f124f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce187e9eb654c958d528ee68e7f124f" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 33691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9a389e1db9d43bf8aa4a7b40279131e" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2adf6ed92b7945ee812b2f165f3e3662" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 37627 } health_report { overall_health: UNKNOWN } } }
I20260924 07:22:17.491660 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.489506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.3.246.66:33691)
0924 07:22:17.489545 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:17.489547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:17.489555 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:22:17.489619 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:17.489620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:17.489633 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:17.489753 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:17.489858 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.490277 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.490956 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.491124 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.491172 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.491334 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:17.491429 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:17.491490 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:17.491585 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 07:22:17.491601 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:22:17.493273 23368 raft_consensus.cc:1261] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662 [term 1 FOLLOWER]: Refusing update from remote peer cce187e9eb654c958d528ee68e7f124f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:17.493296 23636 raft_consensus.cc:1261] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e [term 1 FOLLOWER]: Refusing update from remote peer cce187e9eb654c958d528ee68e7f124f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:17.493609 23707 consensus_queue.cc:1035] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2adf6ed92b7945ee812b2f165f3e3662" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:17.493757 23709 consensus_queue.cc:1035] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a389e1db9d43bf8aa4a7b40279131e" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 38137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:17.497419 23713 mvcc.cc:205] Tried to move back new op lower bound from 7332800665570287616 to 7332800665542451200. Current Snapshot: MvccSnapshot[applied={T|T < 7332800665570287616}]
I20260924 07:22:17.497602 23715 mvcc.cc:205] Tried to move back new op lower bound from 7332800665570287616 to 7332800665542451200. Current Snapshot: MvccSnapshot[applied={T|T < 7332800665570287616}]
I20260924 07:22:17.498560 23223 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:22:17.498657 23223 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:22:17.499372 23223 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a015e1d7bb0848599d2d232ef4e77d9f: Sending DeleteTablet for 3 replicas of tablet 0a9f2ff7879f41aa80a46ea0d48e40b5
I20260924 07:22:17.499682 23482 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9f2ff7879f41aa80a46ea0d48e40b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:17 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:52574
I20260924 07:22:17.499702 23616 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9f2ff7879f41aa80a46ea0d48e40b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:17 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:45058
I20260924 07:22:17.499779 23348 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9f2ff7879f41aa80a46ea0d48e40b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:17 UTC) from {username='kudu', principal='kudu/127.3.246.126@KRBTEST.COM'} at 127.0.0.1:53342
I20260924 07:22:17.499907 23721 tablet_replica.cc:330] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f: stopping tablet replica
I20260924 07:22:17.499977 23723 tablet_replica.cc:330] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662: stopping tablet replica
I20260924 07:22:17.499997 23721 raft_consensus.cc:2229] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:22:17.500073 23723 raft_consensus.cc:2229] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:22:17.500109 23721 raft_consensus.cc:2258] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:17.500180 23723 raft_consensus.cc:2258] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P 2adf6ed92b7945ee812b2f165f3e3662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:17.500259 23722 tablet_replica.cc:330] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e: stopping tablet replica
I20260924 07:22:17.500345 23722 raft_consensus.cc:2229] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:22:17.500437 23722 raft_consensus.cc:2258] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:17.500497 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 23285
I20260924 07:22:17.501361 23721 ts_tablet_manager.cc:1905] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:22:17.501392 23722 ts_tablet_manager.cc:1905] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:22:17.503304 23721 ts_tablet_manager.cc:1918] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:22:17.503293 23722 ts_tablet_manager.cc:1918] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:22:17.503376 23721 log.cc:1196] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/0a9f2ff7879f41aa80a46ea0d48e40b5
I20260924 07:22:17.503490 23722 log.cc:1196] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/0a9f2ff7879f41aa80a46ea0d48e40b5
I20260924 07:22:17.503659 23721 ts_tablet_manager.cc:1939] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P cce187e9eb654c958d528ee68e7f124f: Deleting consensus metadata
I20260924 07:22:17.503962 23722 ts_tablet_manager.cc:1939] T 0a9f2ff7879f41aa80a46ea0d48e40b5 P f9a389e1db9d43bf8aa4a7b40279131e: Deleting consensus metadata
I20260924 07:22:17.504137 23209 catalog_manager.cc:4841] TS cce187e9eb654c958d528ee68e7f124f (127.3.246.66:33691): tablet 0a9f2ff7879f41aa80a46ea0d48e40b5 (table test-table [id=87624f12f1ac478d81d34d86dbb11ce8]) successfully deleted
I20260924 07:22:17.504417 23210 catalog_manager.cc:4841] TS f9a389e1db9d43bf8aa4a7b40279131e (127.3.246.67:38137): tablet 0a9f2ff7879f41aa80a46ea0d48e40b5 (table test-table [id=87624f12f1ac478d81d34d86dbb11ce8]) successfully deleted
W20260924 07:22:17.508628 23210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:37627 (error 108)
I20260924 07:22:17.509164 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.509030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:37627 (local address 127.0.0.1:53356)
0924 07:22:17.509086 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:17.509139 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:37627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 07:22:17.509193 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 23419
W20260924 07:22:17.510084 23210 catalog_manager.cc:4584] TS 2adf6ed92b7945ee812b2f165f3e3662 (127.3.246.65:37627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a9f2ff7879f41aa80a46ea0d48e40b5: Network error: Client connection negotiation failed: client connection to 127.3.246.65:37627: connect: Connection refused (error 111)
I20260924 07:22:17.513962 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 23553
I20260924 07:22:17.518296 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 23191
2026-09-24T07:22:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1924 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T07:22:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:17Z Disabled control of system clock
I20260924 07:22:17.545799 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:34901
--webserver_interface=127.3.246.126
--webserver_port=0
--builtin_ntp_servers=127.3.246.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:34901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:22:17.626015 23728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:17.626232 23728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:17.626252 23728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:17.627475 23728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:17.627517 23728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:17.627530 23728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:17.627543 23728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:17.627604 23728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:17.628947 23728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:34901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:34901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:17.629194 23728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:17.629498 23728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:17.634023 23736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:17.634073 23733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:17.634091 23734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:17.634213 23728 server_base.cc:1034] running on GCE node
I20260924 07:22:17.634485 23728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:17.634757 23728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:17.635921 23728 hybrid_clock.cc:648] HybridClock initialized: now 1790234537635915 us; error 35 us; skew 500 ppm
I20260924 07:22:17.637142 23728 webserver.cc:460] Webserver started at http://127.3.246.126:45419/ using document root <none> and password file <none>
I20260924 07:22:17.637379 23728 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:17.637452 23728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:17.637619 23728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:17.638638 23728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "ed3a00ca6c96435bb1d5a1e110c714e7"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "277fb561bfd855f2a2b58739e06c71d0"
server_key_iv: "56f0d53d5190bef905fd01c1283019ea"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.638990 23728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "ed3a00ca6c96435bb1d5a1e110c714e7"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "277fb561bfd855f2a2b58739e06c71d0"
server_key_iv: "56f0d53d5190bef905fd01c1283019ea"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.640270 23728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:22:17.641057 23742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.641238 23728 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:17.641335 23728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "ed3a00ca6c96435bb1d5a1e110c714e7"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "277fb561bfd855f2a2b58739e06c71d0"
server_key_iv: "56f0d53d5190bef905fd01c1283019ea"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.641415 23728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:17.652801 23728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:17.653083 23728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:17.653236 23728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:17.657133 23728 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:34901
I20260924 07:22:17.657164 23794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:34901 every 8 connection(s)
I20260924 07:22:17.657498 23728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:17.658524 23795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:17.660066 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 23728
I20260924 07:22:17.660202 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:17.660403 4057 external_mini_cluster.cc:1428] Setting key 0d559f4b95f27fd8889fad13ca465bfa
I20260924 07:22:17.660982 23795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7: Bootstrap starting.
I20260924 07:22:17.661707 23795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:17.662043 23795 log.cc:824] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:17.662802 23795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7: No bootstrap required, opened a new log
I20260924 07:22:17.664943 23795 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3a00ca6c96435bb1d5a1e110c714e7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34901 } }
I20260924 07:22:17.665139 23795 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:17.665177 23795 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3a00ca6c96435bb1d5a1e110c714e7, State: Initialized, Role: FOLLOWER
I20260924 07:22:17.665318 23795 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3a00ca6c96435bb1d5a1e110c714e7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34901 } }
I20260924 07:22:17.665416 23795 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:17.665452 23795 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:17.665498 23795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:17.666234 23795 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3a00ca6c96435bb1d5a1e110c714e7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34901 } }
I20260924 07:22:17.666342 23795 leader_election.cc:302] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed3a00ca6c96435bb1d5a1e110c714e7; no voters:
I20260924 07:22:17.666515 23795 leader_election.cc:288] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:17.666590 23800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:17.666751 23800 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [term 1 LEADER]: Becoming Leader. State: Replica: ed3a00ca6c96435bb1d5a1e110c714e7, State: Running, Role: LEADER
I20260924 07:22:17.666776 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.660917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.3.246.126:34901)
0924 07:22:17.661125 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:17.661134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:17.661165 (+ 31us) server_negotiation.cc:407] Connection header received
0924 07:22:17.661799 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:17.661809 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:17.661905 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:17.662063 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:17.662291 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.663602 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.664353 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.665016 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.665142 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.665883 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:17.665887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:17.665887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:17.666091 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:17.666138 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:17.666153 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:17.666237 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:17.666293 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 07:22:17.666481 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260924 07:22:17.666831 23800 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3a00ca6c96435bb1d5a1e110c714e7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34901 } }
I20260924 07:22:17.666779 23795 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:17.667343 23801 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed3a00ca6c96435bb1d5a1e110c714e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3a00ca6c96435bb1d5a1e110c714e7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34901 } } }
I20260924 07:22:17.667428 23801 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:17.667568 23802 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed3a00ca6c96435bb1d5a1e110c714e7. Latest consensus state: current_term: 1 leader_uuid: "ed3a00ca6c96435bb1d5a1e110c714e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3a00ca6c96435bb1d5a1e110c714e7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 34901 } } }
I20260924 07:22:17.667639 23802 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:17.667742 23804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:17.668180 23804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:17.669785 23804 catalog_manager.cc:1347] Generated new cluster ID: 70939bee553447c29756e8910686abb7
I20260924 07:22:17.669839 23804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:17.678190 23804 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:17.678684 23804 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:17.690920 23804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7: Generated new TSK 0
I20260924 07:22:17.691133 23804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:17.696902 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.65:0
--local_ip_for_outbound_sockets=127.3.246.65
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34901
--builtin_ntp_servers=127.3.246.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:22:17.777819 23819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:17.778009 23819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:17.778030 23819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:17.779158 23819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:17.779196 23819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:17.779220 23819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.65
I20260924 07:22:17.780529 23819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.246.65
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.65
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:17.780773 23819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:17.781070 23819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:17.786984 23824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:17.787019 23825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:17.787137 23827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:17.787285 23819 server_base.cc:1034] running on GCE node
I20260924 07:22:17.787493 23819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:17.787721 23819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:17.788883 23819 hybrid_clock.cc:648] HybridClock initialized: now 1790234537788867 us; error 30 us; skew 500 ppm
I20260924 07:22:17.790161 23819 webserver.cc:460] Webserver started at http://127.3.246.65:40717/ using document root <none> and password file <none>
I20260924 07:22:17.790401 23819 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:17.790469 23819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:17.790609 23819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:17.791718 23819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/data/instance:
uuid: "e4da5e4316c84b149ba15759931c8551"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "187917d7f6d7ee758631ad371cd3f653"
server_key_iv: "f95caba9e1c5caa243f25d7c01167259"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.792083 23819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance:
uuid: "e4da5e4316c84b149ba15759931c8551"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "187917d7f6d7ee758631ad371cd3f653"
server_key_iv: "f95caba9e1c5caa243f25d7c01167259"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.793408 23819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:22:17.794250 23833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.794427 23819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:22:17.794520 23819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/wal
uuid: "e4da5e4316c84b149ba15759931c8551"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "187917d7f6d7ee758631ad371cd3f653"
server_key_iv: "f95caba9e1c5caa243f25d7c01167259"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.794601 23819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:17.807173 23819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:17.807442 23819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:17.807585 23819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:17.807788 23819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:17.808076 23819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:17.808126 23819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.808166 23819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:17.808192 23819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.813764 23819 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.65:46077
I20260924 07:22:17.813824 23946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.65:46077 every 8 connection(s)
I20260924 07:22:17.814143 23819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/data/info.pb
I20260924 07:22:17.819342 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.814862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:50349 (local address 127.3.246.126:34901)
0924 07:22:17.814902 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:17.814904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:17.815148 (+ 244us) server_negotiation.cc:407] Connection header received
0924 07:22:17.815606 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:17.815607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:17.815616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:17.815673 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:17.816415 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.816929 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.818029 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.818237 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.818302 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.819104 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:17.819105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:17.819106 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:17.819110 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:17.819125 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:17.819128 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:17.819178 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:17.819258 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 07:22:17.819284 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:22:17.819506 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.814835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:34901 (local address 127.3.246.65:50349)
0924 07:22:17.815021 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:17.815061 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:17.815519 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:17.815737 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:17.815739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:17.815747 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:17.816303 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.816309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.817066 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.817068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.817937 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.817940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.818328 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.818329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.818440 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.818972 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:22:17.818972 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:17.818998 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:22:17.819020 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:17.819188 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:17.819192 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:17.819194 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:22:17.819244 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 07:22:17.819320 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260924 07:22:17.820005 23947 heartbeater.cc:347] Connected to a master server at 127.3.246.126:34901
I20260924 07:22:17.820089 23947 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:17.820225 23947 heartbeater.cc:511] Master 127.3.246.126:34901 requested a full tablet report, sending...
I20260924 07:22:17.820703 23759 ts_manager.cc:194] Registered new tserver with Master: e4da5e4316c84b149ba15759931c8551 (127.3.246.65:46077)
I20260924 07:22:17.821378 23759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.246.65:50349
I20260924 07:22:17.821503 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 23819
I20260924 07:22:17.821612 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-0/wal/instance
I20260924 07:22:17.821875 4057 external_mini_cluster.cc:1428] Setting key 32533dfddcfdc45fac1b871d36f9dc79
I20260924 07:22:17.822863 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.66:0
--local_ip_for_outbound_sockets=127.3.246.66
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34901
--builtin_ntp_servers=127.3.246.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:22:17.902809 23950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:17.903004 23950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:17.903026 23950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:17.904278 23950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:17.904321 23950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:17.904347 23950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.66
I20260924 07:22:17.905755 23950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.246.66
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.66
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:17.905984 23950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:17.906284 23950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:17.911252 23956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:17.911260 23958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:17.911259 23955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:17.911607 23950 server_base.cc:1034] running on GCE node
I20260924 07:22:17.911787 23950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:17.911967 23950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:17.913108 23950 hybrid_clock.cc:648] HybridClock initialized: now 1790234537913095 us; error 28 us; skew 500 ppm
I20260924 07:22:17.914336 23950 webserver.cc:460] Webserver started at http://127.3.246.66:41305/ using document root <none> and password file <none>
I20260924 07:22:17.914563 23950 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:17.914608 23950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:17.914780 23950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:17.915901 23950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/data/instance:
uuid: "08c8f68395e2400098f7f062e5ba701c"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "53a2a8872cd83ed066f032fb627ecc99"
server_key_iv: "a9ffe5862eb1e8e79f036bd2ec8f265b"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.916250 23950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance:
uuid: "08c8f68395e2400098f7f062e5ba701c"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "53a2a8872cd83ed066f032fb627ecc99"
server_key_iv: "a9ffe5862eb1e8e79f036bd2ec8f265b"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.917478 23950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:22:17.918329 23964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.918537 23950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:22:17.918633 23950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/wal
uuid: "08c8f68395e2400098f7f062e5ba701c"
format_stamp: "Formatted at 2026-09-24 07:22:17 on dist-test-slave-rc1w"
server_key: "53a2a8872cd83ed066f032fb627ecc99"
server_key_iv: "a9ffe5862eb1e8e79f036bd2ec8f265b"
server_key_version: "encryptionkey@0"
I20260924 07:22:17.918727 23950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:17.933523 23950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:17.933890 23950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:17.934038 23950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:17.934258 23950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:17.934567 23950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:17.934621 23950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.934674 23950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:17.934710 23950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:17.940706 23950 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.66:41471
I20260924 07:22:17.940748 24077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.66:41471 every 8 connection(s)
I20260924 07:22:17.941056 23950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/data/info.pb
I20260924 07:22:17.946519 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.941748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.66:44823 (local address 127.3.246.126:34901)
0924 07:22:17.941791 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:17.941794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:17.942155 (+ 361us) server_negotiation.cc:407] Connection header received
0924 07:22:17.942678 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:17.942679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:17.942692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:17.942753 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:17.943412 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.943997 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.945043 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.945251 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.945339 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.946210 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:17.946211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:17.946212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:17.946216 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:17.946231 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:17.946235 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:17.946308 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:17.946403 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 07:22:17.946432 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:22:17.946717 24080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:17.941749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:34901 (local address 127.3.246.66:44823)
0924 07:22:17.942033 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:17.942066 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:17.942573 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:17.942827 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:17.942831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:17.942838 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:17.943317 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.943321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.944099 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.944122 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.944913 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:17.944915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:17.945348 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:17.945349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:17.945481 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:17.946055 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:22:17.946056 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:17.946083 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:22:17.946104 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:17.946312 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:17.946317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:17.946321 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:17.946404 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:22:17.946501 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":127,"threads_started":1}
I20260924 07:22:17.947230 24078 heartbeater.cc:347] Connected to a master server at 127.3.246.126:34901
I20260924 07:22:17.947329 24078 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:17.947412 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 23950
I20260924 07:22:17.947520 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/wal/instance
I20260924 07:22:17.947530 24078 heartbeater.cc:511] Master 127.3.246.126:34901 requested a full tablet report, sending...
I20260924 07:22:17.947736 4057 external_mini_cluster.cc:1428] Setting key 798882ad06f214fa4cda18d14854e6b3
I20260924 07:22:17.947902 23759 ts_manager.cc:194] Registered new tserver with Master: 08c8f68395e2400098f7f062e5ba701c (127.3.246.66:41471)
I20260924 07:22:17.948558 23759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.246.66:44823
I20260924 07:22:17.948938 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.246.67:0
--local_ip_for_outbound_sockets=127.3.246.67
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34901
--builtin_ntp_servers=127.3.246.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:22:18.028321 24081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:18.028517 24081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:18.028538 24081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:18.029692 24081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:18.029731 24081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:18.029755 24081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.246.67
I20260924 07:22:18.031064 24081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.246.67
--webserver_port=0
--tserver_master_addrs=127.3.246.126:34901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.246.67
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:18.031325 24081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:18.031620 24081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:18.036531 24087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:18.036517 24089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:18.036530 24086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:18.036849 24081 server_base.cc:1034] running on GCE node
I20260924 07:22:18.037106 24081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:18.037354 24081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:18.038523 24081 hybrid_clock.cc:648] HybridClock initialized: now 1790234538038516 us; error 35 us; skew 500 ppm
I20260924 07:22:18.039834 24081 webserver.cc:460] Webserver started at http://127.3.246.67:44597/ using document root <none> and password file <none>
I20260924 07:22:18.040072 24081 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:18.040143 24081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:18.040277 24081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:18.041311 24081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/data/instance:
uuid: "0a558b297215424c8d3b9c677c7da609"
format_stamp: "Formatted at 2026-09-24 07:22:18 on dist-test-slave-rc1w"
server_key: "8bf6100204fbbb667c0a5556bf45b92d"
server_key_iv: "b2ac7333cdc0590f2ea3d0ba0760be16"
server_key_version: "encryptionkey@0"
I20260924 07:22:18.041723 24081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance:
uuid: "0a558b297215424c8d3b9c677c7da609"
format_stamp: "Formatted at 2026-09-24 07:22:18 on dist-test-slave-rc1w"
server_key: "8bf6100204fbbb667c0a5556bf45b92d"
server_key_iv: "b2ac7333cdc0590f2ea3d0ba0760be16"
server_key_version: "encryptionkey@0"
I20260924 07:22:18.042975 24081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:22:18.043785 24095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:18.043989 24081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:22:18.044085 24081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/wal
uuid: "0a558b297215424c8d3b9c677c7da609"
format_stamp: "Formatted at 2026-09-24 07:22:18 on dist-test-slave-rc1w"
server_key: "8bf6100204fbbb667c0a5556bf45b92d"
server_key_iv: "b2ac7333cdc0590f2ea3d0ba0760be16"
server_key_version: "encryptionkey@0"
I20260924 07:22:18.044173 24081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:18.057348 24081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:18.057639 24081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:18.057804 24081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:18.058022 24081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:22:18.058329 24081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:22:18.058384 24081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:18.058418 24081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:22:18.058444 24081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:18.064096 24081 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.67:46279
I20260924 07:22:18.064134 24208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.67:46279 every 8 connection(s)
I20260924 07:22:18.064481 24081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/data/info.pb
I20260924 07:22:18.069869 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.065186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.67:56091 (local address 127.3.246.126:34901)
0924 07:22:18.065235 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:18.065237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:18.065479 (+ 242us) server_negotiation.cc:407] Connection header received
0924 07:22:18.066081 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:18.066082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:18.066095 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:18.066156 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:18.066923 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.067414 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.068471 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.068667 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.068778 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.069550 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:18.069552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:18.069552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:18.069557 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:18.069570 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:18.069573 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:18.069640 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:18.069763 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 07:22:18.069788 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:22:18.070108 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.065167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:34901 (local address 127.3.246.67:56091)
0924 07:22:18.065364 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:18.065396 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:18.065973 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:18.066247 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:18.066251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:18.066266 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:18.066824 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:18.066829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.067523 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.067524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:18.068363 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:18.068366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.068785 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.068786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:18.068927 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.069406 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:22:18.069407 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:18.069433 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:22:18.069454 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:18.069641 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:18.069647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:18.069650 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:22:18.069762 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 07:22:18.069860 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260924 07:22:18.070612 24209 heartbeater.cc:347] Connected to a master server at 127.3.246.126:34901
I20260924 07:22:18.070684 24209 heartbeater.cc:464] Registering TS with master...
I20260924 07:22:18.070874 24209 heartbeater.cc:511] Master 127.3.246.126:34901 requested a full tablet report, sending...
I20260924 07:22:18.071250 23759 ts_manager.cc:194] Registered new tserver with Master: 0a558b297215424c8d3b9c677c7da609 (127.3.246.67:46279)
I20260924 07:22:18.071599 23759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.246.67:56091
I20260924 07:22:18.073639 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 24081
I20260924 07:22:18.073792 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/wal/instance
I20260924 07:22:18.074004 4057 external_mini_cluster.cc:1428] Setting key a1dc3a282ed1914c56207f7c956f9307
I20260924 07:22:18.074863 4057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:22:18.079003 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.076291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.3.246.126:34901)
0924 07:22:18.076345 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:18.076347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:18.076437 (+ 90us) server_negotiation.cc:407] Connection header received
0924 07:22:18.076486 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:18.076486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:18.076494 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:18.076536 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:18.076728 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.077195 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.077976 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.078157 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.078219 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.078848 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:18.078849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:18.078849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:18.078852 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:18.078865 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:18.078868 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:18.078905 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:18.078940 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 07:22:18.078955 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:22:18.080595 23759 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:22:18.082237 23759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:22:18.088387 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.084167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:46279 (local address 127.0.0.1:33058)
0924 07:22:18.084536 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:18.084544 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:18.084583 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:18.084744 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:18.084747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:18.084749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:18.084839 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:18.084841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.086221 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.086223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:18.087248 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:18.087251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.088075 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.088076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:18.088150 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.088165 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 07:22:18.088243 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:22:18.088295 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":59,"thread_start_us":94,"threads_started":1}
I20260924 07:22:18.088507 24219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.084131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:41471 (local address 127.0.0.1:51300)
0924 07:22:18.084328 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:18.084367 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:18.084435 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:18.085083 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:18.085084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:18.085086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:18.085137 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:18.085139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.086200 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.086201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:18.087522 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:18.087525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.088261 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.088262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:18.088325 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.088329 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:18.088391 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 07:22:18.088429 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":141,"thread_start_us":57,"threads_started":1}
I20260924 07:22:18.088696 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.084167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:46077 (local address 127.0.0.1:57162)
0924 07:22:18.084398 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:18.084406 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:18.084448 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:18.084744 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:18.084747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:18.084749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:18.084839 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:18.084841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.086249 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.086250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:18.087522 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:18.087525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.088479 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.088480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:18.088538 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.088542 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:18.088593 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:22:18.088632 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":58,"thread_start_us":107,"threads_started":1}
I20260924 07:22:18.088706 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.084393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.3.246.67:46279)
0924 07:22:18.084556 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:18.084560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:18.084574 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:22:18.084637 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:18.084639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:18.084651 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:18.084736 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:18.085037 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.086117 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.087332 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.087975 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.088050 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.088133 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:18.088586 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 07:22:18.088625 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260924 07:22:18.088935 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.084408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.3.246.65:46077)
0924 07:22:18.084569 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:18.084573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:18.084583 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:22:18.084622 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:18.084624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:18.084640 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:18.084734 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:18.084931 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.086195 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.087613 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.088299 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.088365 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.088446 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:18.088814 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 07:22:18.088855 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260924 07:22:18.089429 24143 tablet_service.cc:1467] Processing CreateTablet for tablet e515df21471e4a6a9d4a5350c112e4f2 (DEFAULT_TABLE table=test-table [id=18213365d32a4571bcec9dd74982d789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:18.089529 23881 tablet_service.cc:1467] Processing CreateTablet for tablet e515df21471e4a6a9d4a5350c112e4f2 (DEFAULT_TABLE table=test-table [id=18213365d32a4571bcec9dd74982d789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:18.089723 24220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.084215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.3.246.66:41471)
0924 07:22:18.084858 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:18.084861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:18.084868 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:22:18.084925 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:18.084927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:18.084941 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:18.085015 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:18.085224 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.086099 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.087627 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.088173 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.088250 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.088332 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:18.088405 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 07:22:18.089111 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":100,"threads_started":1}
I20260924 07:22:18.089838 24143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e515df21471e4a6a9d4a5350c112e4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:18.089838 23881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e515df21471e4a6a9d4a5350c112e4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:18.090265 24012 tablet_service.cc:1467] Processing CreateTablet for tablet e515df21471e4a6a9d4a5350c112e4f2 (DEFAULT_TABLE table=test-table [id=18213365d32a4571bcec9dd74982d789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:22:18.090566 24012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e515df21471e4a6a9d4a5350c112e4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:18.093089 24223 tablet_bootstrap.cc:492] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c: Bootstrap starting.
I20260924 07:22:18.093436 24224 tablet_bootstrap.cc:492] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551: Bootstrap starting.
I20260924 07:22:18.093434 24225 tablet_bootstrap.cc:492] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609: Bootstrap starting.
I20260924 07:22:18.093816 24223 tablet_bootstrap.cc:654] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:18.094111 24225 tablet_bootstrap.cc:654] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:18.094111 24223 log.cc:824] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:18.094129 24224 tablet_bootstrap.cc:654] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:18.094456 24225 log.cc:824] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:18.094456 24224 log.cc:824] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:18.094892 24223 tablet_bootstrap.cc:492] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c: No bootstrap required, opened a new log
I20260924 07:22:18.094962 24223 ts_tablet_manager.cc:1397] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:22:18.095206 24225 tablet_bootstrap.cc:492] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609: No bootstrap required, opened a new log
I20260924 07:22:18.095285 24225 ts_tablet_manager.cc:1397] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:22:18.097018 24223 raft_consensus.cc:348] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c8f68395e2400098f7f062e5ba701c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41471 } } peers { permanent_uuid: "e4da5e4316c84b149ba15759931c8551" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46077 } } peers { permanent_uuid: "0a558b297215424c8d3b9c677c7da609" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 46279 } }
I20260924 07:22:18.097149 24223 raft_consensus.cc:374] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:18.097173 24223 raft_consensus.cc:729] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c8f68395e2400098f7f062e5ba701c, State: Initialized, Role: FOLLOWER
I20260924 07:22:18.097110 24225 raft_consensus.cc:348] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c8f68395e2400098f7f062e5ba701c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41471 } } peers { permanent_uuid: "e4da5e4316c84b149ba15759931c8551" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46077 } } peers { permanent_uuid: "0a558b297215424c8d3b9c677c7da609" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 46279 } }
I20260924 07:22:18.097293 24223 consensus_queue.cc:260] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c8f68395e2400098f7f062e5ba701c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41471 } } peers { permanent_uuid: "e4da5e4316c84b149ba15759931c8551" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46077 } } peers { permanent_uuid: "0a558b297215424c8d3b9c677c7da609" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 46279 } }
I20260924 07:22:18.097326 24225 raft_consensus.cc:374] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:18.097369 24225 raft_consensus.cc:729] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a558b297215424c8d3b9c677c7da609, State: Initialized, Role: FOLLOWER
I20260924 07:22:18.097536 24223 ts_tablet_manager.cc:1428] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:22:18.097648 24078 heartbeater.cc:503] Master 127.3.246.126:34901 was elected leader, sending a full tablet report...
I20260924 07:22:18.097836 24224 tablet_bootstrap.cc:492] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551: No bootstrap required, opened a new log
I20260924 07:22:18.097915 24224 ts_tablet_manager.cc:1397] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 07:22:18.099915 24224 raft_consensus.cc:348] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c8f68395e2400098f7f062e5ba701c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41471 } } peers { permanent_uuid: "e4da5e4316c84b149ba15759931c8551" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46077 } } peers { permanent_uuid: "0a558b297215424c8d3b9c677c7da609" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 46279 } }
I20260924 07:22:18.100041 24224 raft_consensus.cc:374] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:18.100062 24224 raft_consensus.cc:729] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4da5e4316c84b149ba15759931c8551, State: Initialized, Role: FOLLOWER
I20260924 07:22:18.100168 24224 consensus_queue.cc:260] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c8f68395e2400098f7f062e5ba701c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41471 } } peers { permanent_uuid: "e4da5e4316c84b149ba15759931c8551" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46077 } } peers { permanent_uuid: "0a558b297215424c8d3b9c677c7da609" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 46279 } }
I20260924 07:22:18.100389 24224 ts_tablet_manager.cc:1428] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 07:22:18.100471 23947 heartbeater.cc:503] Master 127.3.246.126:34901 was elected leader, sending a full tablet report...
I20260924 07:22:18.100529 24225 consensus_queue.cc:260] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c8f68395e2400098f7f062e5ba701c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41471 } } peers { permanent_uuid: "e4da5e4316c84b149ba15759931c8551" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46077 } } peers { permanent_uuid: "0a558b297215424c8d3b9c677c7da609" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 46279 } }
I20260924 07:22:18.100819 24225 ts_tablet_manager.cc:1428] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260924 07:22:18.100879 24209 heartbeater.cc:503] Master 127.3.246.126:34901 was elected leader, sending a full tablet report...
W20260924 07:22:18.191557 24079 tablet.cc:2369] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:22:18.314836 23948 tablet.cc:2369] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:22:18.315047 24210 tablet.cc:2369] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:22:18.340927 24230 raft_consensus.cc:482] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:22:18.341079 24230 raft_consensus.cc:504] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c8f68395e2400098f7f062e5ba701c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41471 } } peers { permanent_uuid: "e4da5e4316c84b149ba15759931c8551" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46077 } } peers { permanent_uuid: "0a558b297215424c8d3b9c677c7da609" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 46279 } }
I20260924 07:22:18.341564 24230 leader_election.cc:288] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08c8f68395e2400098f7f062e5ba701c (127.3.246.66:41471), 0a558b297215424c8d3b9c677c7da609 (127.3.246.67:46279)
I20260924 07:22:18.344599 24233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.341796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.67:46279 (local address 127.3.246.65:47139)
0924 07:22:18.341943 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:18.341955 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:18.342027 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:18.342159 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:18.342159 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:18.342161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:18.342250 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:18.342252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.342892 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.342893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:18.343666 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:18.343668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.344310 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.344311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:18.344381 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.344385 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:18.344462 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:22:18.344505 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260924 07:22:18.344591 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.341907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:47139 (local address 127.3.246.67:46279)
0924 07:22:18.341979 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:18.341981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:18.342009 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:22:18.342096 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:18.342097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:18.342105 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:18.342157 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:18.342326 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.342801 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.343770 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.344196 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.344307 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.344316 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:18.344490 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 07:22:18.344534 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:22:18.344707 24220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.341815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.246.65:35909 (local address 127.3.246.66:41471)
0924 07:22:18.341866 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:18.341869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:18.341964 (+ 95us) server_negotiation.cc:407] Connection header received
0924 07:22:18.342097 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:18.342098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:18.342109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:18.342200 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:22:18.342369 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.342854 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.343901 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.344303 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.344426 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.344435 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:18.344602 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 07:22:18.344640 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:22:18.344723 24232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.341751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.66:41471 (local address 127.3.246.65:35909)
0924 07:22:18.341873 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:18.341887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:18.342007 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:18.342204 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:18.342205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:18.342207 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:22:18.342292 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:18.342294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.342946 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.342948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:18.343779 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:18.343781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.344413 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.344414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:18.344497 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.344501 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:22:18.344593 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 07:22:18.344633 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260924 07:22:18.345160 24032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e515df21471e4a6a9d4a5350c112e4f2" candidate_uuid: "e4da5e4316c84b149ba15759931c8551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c8f68395e2400098f7f062e5ba701c" is_pre_election: true
I20260924 07:22:18.345331 24032 raft_consensus.cc:2454] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4da5e4316c84b149ba15759931c8551 in term 0.
I20260924 07:22:18.345326 24163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e515df21471e4a6a9d4a5350c112e4f2" candidate_uuid: "e4da5e4316c84b149ba15759931c8551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a558b297215424c8d3b9c677c7da609" is_pre_election: true
I20260924 07:22:18.345449 24163 raft_consensus.cc:2454] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4da5e4316c84b149ba15759931c8551 in term 0.
I20260924 07:22:18.345556 23836 leader_election.cc:302] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c8f68395e2400098f7f062e5ba701c, e4da5e4316c84b149ba15759931c8551; no voters:
I20260924 07:22:18.345712 24230 raft_consensus.cc:2781] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:22:18.345788 24230 raft_consensus.cc:482] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:22:18.345805 24230 raft_consensus.cc:3037] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:18.346536 24230 raft_consensus.cc:504] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c8f68395e2400098f7f062e5ba701c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41471 } } peers { permanent_uuid: "e4da5e4316c84b149ba15759931c8551" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46077 } } peers { permanent_uuid: "0a558b297215424c8d3b9c677c7da609" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 46279 } }
I20260924 07:22:18.346679 24230 leader_election.cc:288] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [CANDIDATE]: Term 1 election: Requested vote from peers 08c8f68395e2400098f7f062e5ba701c (127.3.246.66:41471), 0a558b297215424c8d3b9c677c7da609 (127.3.246.67:46279)
I20260924 07:22:18.346823 24032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e515df21471e4a6a9d4a5350c112e4f2" candidate_uuid: "e4da5e4316c84b149ba15759931c8551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c8f68395e2400098f7f062e5ba701c"
I20260924 07:22:18.346848 24163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e515df21471e4a6a9d4a5350c112e4f2" candidate_uuid: "e4da5e4316c84b149ba15759931c8551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a558b297215424c8d3b9c677c7da609"
I20260924 07:22:18.346922 24032 raft_consensus.cc:3037] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:18.346917 24163 raft_consensus.cc:3037] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:18.347677 24163 raft_consensus.cc:2454] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4da5e4316c84b149ba15759931c8551 in term 1.
I20260924 07:22:18.347700 24032 raft_consensus.cc:2454] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4da5e4316c84b149ba15759931c8551 in term 1.
I20260924 07:22:18.347852 23834 leader_election.cc:302] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a558b297215424c8d3b9c677c7da609, e4da5e4316c84b149ba15759931c8551; no voters:
I20260924 07:22:18.347956 24230 raft_consensus.cc:2781] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:18.348133 24230 raft_consensus.cc:686] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [term 1 LEADER]: Becoming Leader. State: Replica: e4da5e4316c84b149ba15759931c8551, State: Running, Role: LEADER
I20260924 07:22:18.348227 24230 consensus_queue.cc:237] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c8f68395e2400098f7f062e5ba701c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41471 } } peers { permanent_uuid: "e4da5e4316c84b149ba15759931c8551" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46077 } } peers { permanent_uuid: "0a558b297215424c8d3b9c677c7da609" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 46279 } }
I20260924 07:22:18.349020 23758 catalog_manager.cc:5495] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4da5e4316c84b149ba15759931c8551 (127.3.246.65). New cstate: current_term: 1 leader_uuid: "e4da5e4316c84b149ba15759931c8551" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c8f68395e2400098f7f062e5ba701c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4da5e4316c84b149ba15759931c8551" member_type: VOTER last_known_addr { host: "127.3.246.65" port: 46077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a558b297215424c8d3b9c677c7da609" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 46279 } health_report { overall_health: UNKNOWN } } }
I20260924 07:22:18.369587 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.367276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.3.246.65:46077)
0924 07:22:18.367362 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:18.367364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:18.367374 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:22:18.367600 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:18.367602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:18.367620 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:18.367682 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:18.367824 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.368269 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.368964 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:18.369136 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:18.369184 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:18.369290 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:18.369379 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:18.369433 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:18.369469 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 07:22:18.369485 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 07:22:18.371301 24163 raft_consensus.cc:1261] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609 [term 1 FOLLOWER]: Refusing update from remote peer e4da5e4316c84b149ba15759931c8551: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:18.371383 24032 raft_consensus.cc:1261] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c [term 1 FOLLOWER]: Refusing update from remote peer e4da5e4316c84b149ba15759931c8551: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:18.371665 24230 consensus_queue.cc:1035] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08c8f68395e2400098f7f062e5ba701c" member_type: VOTER last_known_addr { host: "127.3.246.66" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:18.371840 24234 consensus_queue.cc:1035] T e515df21471e4a6a9d4a5350c112e4f2 P e4da5e4316c84b149ba15759931c8551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a558b297215424c8d3b9c677c7da609" member_type: VOTER last_known_addr { host: "127.3.246.67" port: 46279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:18.374082 24237 mvcc.cc:205] Tried to move back new op lower bound from 7332800669166440448 to 7332800669074874368. Current Snapshot: MvccSnapshot[applied={T|T < 7332800669166440448}]
I20260924 07:22:18.374368 24238 mvcc.cc:205] Tried to move back new op lower bound from 7332800669166440448 to 7332800669074874368. Current Snapshot: MvccSnapshot[applied={T|T < 7332800669166440448}]
I20260924 07:22:18.374881 24239 mvcc.cc:205] Tried to move back new op lower bound from 7332800669166440448 to 7332800669074874368. Current Snapshot: MvccSnapshot[applied={T|T < 7332800669166440448 or (T in {7332800669166440448})}]
I20260924 07:22:18.376708 23758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:22:18.376811 23758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:22:18.377566 23758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed3a00ca6c96435bb1d5a1e110c714e7: Sending DeleteTablet for 3 replicas of tablet e515df21471e4a6a9d4a5350c112e4f2
I20260924 07:22:18.377943 23881 tablet_service.cc:1514] Processing DeleteTablet for tablet e515df21471e4a6a9d4a5350c112e4f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:18 UTC) from {username='slave'} at 127.0.0.1:57162
I20260924 07:22:18.377934 24143 tablet_service.cc:1514] Processing DeleteTablet for tablet e515df21471e4a6a9d4a5350c112e4f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:18 UTC) from {username='slave'} at 127.0.0.1:33058
I20260924 07:22:18.377969 24012 tablet_service.cc:1514] Processing DeleteTablet for tablet e515df21471e4a6a9d4a5350c112e4f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:22:18 UTC) from {username='slave'} at 127.0.0.1:51300
I20260924 07:22:18.378276 24247 tablet_replica.cc:330] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609: stopping tablet replica
I20260924 07:22:18.378419 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 23819
I20260924 07:22:18.379055 24248 tablet_replica.cc:330] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c: stopping tablet replica
I20260924 07:22:18.379154 24248 raft_consensus.cc:2229] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:22:18.379253 24248 raft_consensus.cc:2258] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:18.379395 24247 raft_consensus.cc:2229] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:22:18.379500 24247 raft_consensus.cc:2258] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:22:18.379731 24248 ts_tablet_manager.cc:1905] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:22:18.379959 24247 ts_tablet_manager.cc:1905] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:22:18.381527 24247 ts_tablet_manager.cc:1918] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:22:18.381618 24247 log.cc:1196] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-2/wal/wals/e515df21471e4a6a9d4a5350c112e4f2
I20260924 07:22:18.381654 24248 ts_tablet_manager.cc:1918] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:22:18.381747 24248 log.cc:1196] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c: Deleting WAL directory at /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234301618289-4057-0/minicluster-data/ts-1/wal/wals/e515df21471e4a6a9d4a5350c112e4f2
I20260924 07:22:18.381903 24247 ts_tablet_manager.cc:1939] T e515df21471e4a6a9d4a5350c112e4f2 P 0a558b297215424c8d3b9c677c7da609: Deleting consensus metadata
I20260924 07:22:18.381980 24248 ts_tablet_manager.cc:1939] T e515df21471e4a6a9d4a5350c112e4f2 P 08c8f68395e2400098f7f062e5ba701c: Deleting consensus metadata
I20260924 07:22:18.382287 23743 catalog_manager.cc:4841] TS 0a558b297215424c8d3b9c677c7da609 (127.3.246.67:46279): tablet e515df21471e4a6a9d4a5350c112e4f2 (table test-table [id=18213365d32a4571bcec9dd74982d789]) successfully deleted
W20260924 07:22:18.384698 23746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.246.65:46077 (error 108)
I20260924 07:22:18.384737 23745 catalog_manager.cc:4841] TS 08c8f68395e2400098f7f062e5ba701c (127.3.246.66:41471): tablet e515df21471e4a6a9d4a5350c112e4f2 (table test-table [id=18213365d32a4571bcec9dd74982d789]) successfully deleted
I20260924 07:22:18.384931 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 23950
I20260924 07:22:18.385710 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:18.385016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.65:46077 (local address 127.0.0.1:60406)
0924 07:22:18.385603 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:18.385650 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.65:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564}
W20260924 07:22:18.385846 23746 catalog_manager.cc:4584] TS e4da5e4316c84b149ba15759931c8551 (127.3.246.65:46077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e515df21471e4a6a9d4a5350c112e4f2: Network error: Client connection negotiation failed: client connection to 127.3.246.65:46077: connect: Connection refused (error 111)
I20260924 07:22:18.389850 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 24081
I20260924 07:22:18.394552 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 23728
2026-09-24T07:22:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (875 ms)
[----------] 6 tests from EncryptionPolicyTest (8428 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:18 dist-test-slave-rc1w krb5kdc[24252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:18 dist-test-slave-rc1w krb5kdc[24252](info): set up 2 sockets
Sep 24 07:22:18 dist-test-slave-rc1w krb5kdc[24252](info): commencing operation
krb5kdc: starting...
W20260924 07:22:19.433715 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:19 dist-test-slave-rc1w krb5kdc[24252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:22:19.549039 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:19.629601 24265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:19.629820 24265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:19.629843 24265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:19.631043 24265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:19.631086 24265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:19.631101 24265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:19.631117 24265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:19.631130 24265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:19.632602 24265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34779
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:19.632865 24265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:19.633100 24265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:19.637717 24270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:19.637688 24273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:19.637712 24271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:19.638046 24265 server_base.cc:1034] running on GCE node
I20260924 07:22:19.638283 24265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:19.638552 24265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:19.639701 24265 hybrid_clock.cc:648] HybridClock initialized: now 1790234539639667 us; error 49 us; skew 500 ppm
Sep 24 07:22:19 dist-test-slave-rc1w krb5kdc[24252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:19.642325 24265 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:22:19.642786 24265 webserver.cc:460] Webserver started at http://127.0.0.1:43289/ using document root <none> and password file <none>
I20260924 07:22:19.643035 24265 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:19.643081 24265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:19.643169 24265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:19.644136 24265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "cf41813c32464826b4e5fb52cb775c76"
format_stamp: "Formatted at 2026-09-24 07:22:19 on dist-test-slave-rc1w"
server_key: "80493f54df473f79f0e73129078e3b72"
server_key_iv: "820409df03feaf567c4264c52f07d230"
server_key_version: "encryptionkey@0"
I20260924 07:22:19.644428 24265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "cf41813c32464826b4e5fb52cb775c76"
format_stamp: "Formatted at 2026-09-24 07:22:19 on dist-test-slave-rc1w"
server_key: "80493f54df473f79f0e73129078e3b72"
server_key_iv: "820409df03feaf567c4264c52f07d230"
server_key_version: "encryptionkey@0"
I20260924 07:22:19.645718 24265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:22:19.646476 24280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:19.646658 24265 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:19.646732 24265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "cf41813c32464826b4e5fb52cb775c76"
format_stamp: "Formatted at 2026-09-24 07:22:19 on dist-test-slave-rc1w"
server_key: "80493f54df473f79f0e73129078e3b72"
server_key_iv: "820409df03feaf567c4264c52f07d230"
server_key_version: "encryptionkey@0"
I20260924 07:22:19.646788 24265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:19.664059 24265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:19.666488 24265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:19.666607 24265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:19.670907 24265 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34779
I20260924 07:22:19.670961 24332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34779 every 8 connection(s)
I20260924 07:22:19.671306 24265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:19.672248 24333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:19.673431 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 24265
I20260924 07:22:19.673578 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:19.673816 4057 external_mini_cluster.cc:1428] Setting key aa63157ef56d1553dacd1b032da41158
I20260924 07:22:19.674731 24333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76: Bootstrap starting.
I20260924 07:22:19.675360 24333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:19.675701 24333 log.cc:824] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:19.676470 24333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76: No bootstrap required, opened a new log
I20260924 07:22:19.678637 24333 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf41813c32464826b4e5fb52cb775c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34779 } }
I20260924 07:22:19.678825 24333 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:19.678870 24333 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf41813c32464826b4e5fb52cb775c76, State: Initialized, Role: FOLLOWER
I20260924 07:22:19.679015 24333 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf41813c32464826b4e5fb52cb775c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34779 } }
I20260924 07:22:19.679111 24333 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:19.679167 24333 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:19.679225 24333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:19.679754 24333 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf41813c32464826b4e5fb52cb775c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34779 } }
I20260924 07:22:19.679874 24333 leader_election.cc:302] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf41813c32464826b4e5fb52cb775c76; no voters:
I20260924 07:22:19.680066 24333 leader_election.cc:288] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:19.680091 24338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:19.680202 24338 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [term 1 LEADER]: Becoming Leader. State: Replica: cf41813c32464826b4e5fb52cb775c76, State: Running, Role: LEADER
I20260924 07:22:19.680284 24338 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf41813c32464826b4e5fb52cb775c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34779 } }
I20260924 07:22:19.680339 24333 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:19.680576 24338 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf41813c32464826b4e5fb52cb775c76. Latest consensus state: current_term: 1 leader_uuid: "cf41813c32464826b4e5fb52cb775c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf41813c32464826b4e5fb52cb775c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34779 } } }
I20260924 07:22:19.680646 24338 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:19.680600 24339 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf41813c32464826b4e5fb52cb775c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf41813c32464826b4e5fb52cb775c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34779 } } }
I20260924 07:22:19.680821 24339 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76 [sys.catalog]: This master's current role is: LEADER
Sep 24 07:22:19 dist-test-slave-rc1w krb5kdc[24252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260924 07:22:19.682088 24353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:22:19.682145 24353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:22:19.682256 24343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:19.682449 24343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:19.684116 24343 catalog_manager.cc:1347] Generated new cluster ID: 57e116c1930c4466b48e362e8a8d653c
I20260924 07:22:19.684165 24343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:19.685336 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:19.674430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52395 (local address 127.0.0.1:34779)
0924 07:22:19.674611 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:19.674619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:19.674644 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:22:19.675098 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:19.675105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:19.675153 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:19.675273 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:19.676140 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:19.677115 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:19.678276 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:19.678660 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:19.678736 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:19.681398 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:19.681403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:19.681403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:19.681416 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:19.683616 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:19.684096 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:19.684099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:19.684100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:19.684205 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:19.684462 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:19.684463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:19.684463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:19.684748 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:19.684889 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:19.685044 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 07:22:19.685135 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260924 07:22:19.686430 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:19.694996 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:19.687824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53599 (local address 127.0.0.1:34779)
0924 07:22:19.687858 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:19.687860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:19.687942 (+ 82us) server_negotiation.cc:407] Connection header received
0924 07:22:19.687971 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:19.687972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:19.687982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:19.688051 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:19.688607 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:19.689478 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:19.690481 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:19.690823 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:19.690897 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:19.692217 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:19.692219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:19.692219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:19.692226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:19.693837 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:19.694217 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:19.694218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:19.694218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:19.694277 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:19.694494 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:19.694495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:19.694495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:19.694621 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:19.694741 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:19.694845 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 07:22:19.694906 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:22:19.715253 24343 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:19.716028 24343 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:19.718786 24343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf41813c32464826b4e5fb52cb775c76: Generated new TSK 0
I20260924 07:22:19.718959 24343 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:19.723120 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 24265
W20260924 07:22:19.727519 24250 connection.cc:535] client connection to 127.0.0.1:34779 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34779): Connection reset by peer (error 104)
2026-09-24T07:22:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:19 dist-test-slave-rc1w krb5kdc[24364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:19 dist-test-slave-rc1w krb5kdc[24364](info): set up 2 sockets
Sep 24 07:22:19 dist-test-slave-rc1w krb5kdc[24364](info): commencing operation
krb5kdc: starting...
W20260924 07:22:20.757833 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:20 dist-test-slave-rc1w krb5kdc[24364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:22:20.873081 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:20.955283 24377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:20.955538 24377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:20.955580 24377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:20.956779 24377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:20.956853 24377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:20.956880 24377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:20.956904 24377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:20.956938 24377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:20.958473 24377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:20.958760 24377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:20.959000 24377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:20.964115 24385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:20.964123 24382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:20.964123 24383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:20.964325 24377 server_base.cc:1034] running on GCE node
I20260924 07:22:20.964552 24377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:20.964797 24377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:20.965929 24377 hybrid_clock.cc:648] HybridClock initialized: now 1790234540965909 us; error 36 us; skew 500 ppm
Sep 24 07:22:20 dist-test-slave-rc1w krb5kdc[24364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:20.978202 24377 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:22:20.978608 24377 webserver.cc:460] Webserver started at http://127.0.0.1:45603/ using document root <none> and password file <none>
I20260924 07:22:20.978808 24377 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:20.978849 24377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:20.978938 24377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:20.979957 24377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "4f4fa46aa6044bfe9eeb529cf6a1bfcc"
format_stamp: "Formatted at 2026-09-24 07:22:20 on dist-test-slave-rc1w"
server_key: "2ef49d5a38a99be8bbc0125381d4ed5a"
server_key_iv: "4163ab576c2b340b082afcb382fd85c1"
server_key_version: "encryptionkey@0"
I20260924 07:22:20.980257 24377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "4f4fa46aa6044bfe9eeb529cf6a1bfcc"
format_stamp: "Formatted at 2026-09-24 07:22:20 on dist-test-slave-rc1w"
server_key: "2ef49d5a38a99be8bbc0125381d4ed5a"
server_key_iv: "4163ab576c2b340b082afcb382fd85c1"
server_key_version: "encryptionkey@0"
I20260924 07:22:20.981541 24377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:22:20.982331 24392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:20.982527 24377 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:22:20.982620 24377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "4f4fa46aa6044bfe9eeb529cf6a1bfcc"
format_stamp: "Formatted at 2026-09-24 07:22:20 on dist-test-slave-rc1w"
server_key: "2ef49d5a38a99be8bbc0125381d4ed5a"
server_key_iv: "4163ab576c2b340b082afcb382fd85c1"
server_key_version: "encryptionkey@0"
I20260924 07:22:20.982694 24377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:20.993782 24377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:20.995764 24377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:20.995930 24377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:20.999724 24377 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41981
I20260924 07:22:20.999758 24444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41981 every 8 connection(s)
I20260924 07:22:21.000106 24377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:21.001086 24445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:21.003407 24445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc: Bootstrap starting.
I20260924 07:22:21.003983 24445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:21.004271 24445 log.cc:824] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:21.004935 24445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc: No bootstrap required, opened a new log
I20260924 07:22:21.006567 24445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4fa46aa6044bfe9eeb529cf6a1bfcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41981 } }
I20260924 07:22:21.006700 24445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:21.006726 24445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4fa46aa6044bfe9eeb529cf6a1bfcc, State: Initialized, Role: FOLLOWER
I20260924 07:22:21.006899 24445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4fa46aa6044bfe9eeb529cf6a1bfcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41981 } }
I20260924 07:22:21.006963 24445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:21.007002 24445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:21.007052 24445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:21.007611 24445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4fa46aa6044bfe9eeb529cf6a1bfcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41981 } }
I20260924 07:22:21.007730 24445 leader_election.cc:302] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f4fa46aa6044bfe9eeb529cf6a1bfcc; no voters:
I20260924 07:22:21.007736 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 24377
I20260924 07:22:21.007840 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:21.007889 24445 leader_election.cc:288] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:21.007978 24448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:21.008072 4057 external_mini_cluster.cc:1428] Setting key 04deb7701283b1c291ea3879abfec770
I20260924 07:22:21.008147 24445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:21.008256 24448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [term 1 LEADER]: Becoming Leader. State: Replica: 4f4fa46aa6044bfe9eeb529cf6a1bfcc, State: Running, Role: LEADER
I20260924 07:22:21.008399 24448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4fa46aa6044bfe9eeb529cf6a1bfcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41981 } }
I20260924 07:22:21.009276 24449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f4fa46aa6044bfe9eeb529cf6a1bfcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4fa46aa6044bfe9eeb529cf6a1bfcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41981 } } }
I20260924 07:22:21.009370 24449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:21.009732 24462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:21.009768 24450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f4fa46aa6044bfe9eeb529cf6a1bfcc. Latest consensus state: current_term: 1 leader_uuid: "4f4fa46aa6044bfe9eeb529cf6a1bfcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4fa46aa6044bfe9eeb529cf6a1bfcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41981 } } }
I20260924 07:22:21.009980 24450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:21.010326 24462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:21.013224 24462 catalog_manager.cc:1347] Generated new cluster ID: 4a088066cf274d24af46278acac6fd82
I20260924 07:22:21.013278 24462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:21.014036 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:21.008718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41615 (local address 127.0.0.1:41981)
0924 07:22:21.008886 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:21.008895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:21.008925 (+ 30us) server_negotiation.cc:407] Connection header received
0924 07:22:21.009054 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:21.009061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:21.009116 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:21.009517 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:21.009627 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:21.011101 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:21.011948 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:21.012569 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:21.012681 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:21.013406 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:21.013409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:21.013409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:21.013572 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:21.013602 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:21.013612 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:21.013755 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:21.013845 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 07:22:21.013873 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":94,"threads_started":1}
I20260924 07:22:21.014966 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:21.025614 24462 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:21.026315 24462 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:21.028966 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:21.017961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33833 (local address 127.0.0.1:41981)
0924 07:22:21.017999 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:21.018001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:21.025312 (+ 7311us) server_negotiation.cc:407] Connection header received
0924 07:22:21.025353 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:21.025354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:21.025365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:21.025435 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:21.025788 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:21.026485 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:21.027526 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:21.027839 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:21.027897 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:21.028647 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:21.028649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:21.028650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:21.028654 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:21.028672 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:21.028676 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:21.028740 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:21.028857 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 07:22:21.028871 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:22:21.029009 24462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f4fa46aa6044bfe9eeb529cf6a1bfcc: Generated new TSK 0
I20260924 07:22:21.029165 24462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:21.033301 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 24377
W20260924 07:22:21.037094 24362 connection.cc:535] client connection to 127.0.0.1:41981 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41981): Connection reset by peer (error 104)
2026-09-24T07:22:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:21 dist-test-slave-rc1w krb5kdc[24477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:21 dist-test-slave-rc1w krb5kdc[24477](info): set up 2 sockets
Sep 24 07:22:21 dist-test-slave-rc1w krb5kdc[24477](info): commencing operation
krb5kdc: starting...
W20260924 07:22:22.069780 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:22 dist-test-slave-rc1w krb5kdc[24477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:22:22.184540 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:22.264261 24490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:22.264460 24490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:22.264482 24490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:22.265628 24490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:22.265688 24490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:22.265710 24490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:22.265723 24490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:22.265735 24490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:22.267194 24490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:22.267381 24490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:22.267539 24490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:22.272220 24495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:22.272338 24496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:22.272269 24498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:22.272656 24490 server_base.cc:1034] running on GCE node
I20260924 07:22:22.272886 24490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:22.273175 24490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:22.274319 24490 hybrid_clock.cc:648] HybridClock initialized: now 1790234542274311 us; error 30 us; skew 500 ppm
Sep 24 07:22:22 dist-test-slave-rc1w krb5kdc[24477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:22.276698 24490 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:22:22.277159 24490 webserver.cc:460] Webserver started at http://127.0.0.1:40667/ using document root <none> and password file <none>
I20260924 07:22:22.277390 24490 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:22.277458 24490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:22.277608 24490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:22.278647 24490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "a2b5a56fd7f9436d8e0e965881703a64"
format_stamp: "Formatted at 2026-09-24 07:22:22 on dist-test-slave-rc1w"
server_key: "cf5e0e4bae0475ff5b9094ce442fadbd"
server_key_iv: "f2ed283c5828b00733bb7276c3240c14"
server_key_version: "encryptionkey@0"
I20260924 07:22:22.279012 24490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "a2b5a56fd7f9436d8e0e965881703a64"
format_stamp: "Formatted at 2026-09-24 07:22:22 on dist-test-slave-rc1w"
server_key: "cf5e0e4bae0475ff5b9094ce442fadbd"
server_key_iv: "f2ed283c5828b00733bb7276c3240c14"
server_key_version: "encryptionkey@0"
I20260924 07:22:22.280301 24490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:22:22.280958 24505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:22.281126 24490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:22.281224 24490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "a2b5a56fd7f9436d8e0e965881703a64"
format_stamp: "Formatted at 2026-09-24 07:22:22 on dist-test-slave-rc1w"
server_key: "cf5e0e4bae0475ff5b9094ce442fadbd"
server_key_iv: "f2ed283c5828b00733bb7276c3240c14"
server_key_version: "encryptionkey@0"
I20260924 07:22:22.281306 24490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:22.284571 24490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:22.286409 24490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:22.286563 24490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:22.290735 24490 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34061
I20260924 07:22:22.290773 24557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34061 every 8 connection(s)
I20260924 07:22:22.291096 24490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:22.292052 24558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:22.294354 24558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64: Bootstrap starting.
I20260924 07:22:22.294915 24558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:22.295202 24558 log.cc:824] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:22.295985 24558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64: No bootstrap required, opened a new log
I20260924 07:22:22.297562 24558 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b5a56fd7f9436d8e0e965881703a64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34061 } }
I20260924 07:22:22.297713 24558 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:22.297772 24558 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b5a56fd7f9436d8e0e965881703a64, State: Initialized, Role: FOLLOWER
I20260924 07:22:22.297945 24558 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b5a56fd7f9436d8e0e965881703a64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34061 } }
I20260924 07:22:22.298008 24558 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:22.298066 24558 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:22.298115 24558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:22.298640 24558 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b5a56fd7f9436d8e0e965881703a64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34061 } }
I20260924 07:22:22.298753 24558 leader_election.cc:302] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2b5a56fd7f9436d8e0e965881703a64; no voters:
I20260924 07:22:22.298974 24558 leader_election.cc:288] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:22.298997 24561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:22.299124 24561 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [term 1 LEADER]: Becoming Leader. State: Replica: a2b5a56fd7f9436d8e0e965881703a64, State: Running, Role: LEADER
I20260924 07:22:22.299134 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 24490
I20260924 07:22:22.299243 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:22.299255 24558 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:22.299280 24561 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b5a56fd7f9436d8e0e965881703a64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34061 } }
I20260924 07:22:22.299427 4057 external_mini_cluster.cc:1428] Setting key e5742461842e5fd571babee46e058797
I20260924 07:22:22.299610 24563 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2b5a56fd7f9436d8e0e965881703a64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b5a56fd7f9436d8e0e965881703a64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34061 } } }
I20260924 07:22:22.299690 24563 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:22.299750 24562 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2b5a56fd7f9436d8e0e965881703a64. Latest consensus state: current_term: 1 leader_uuid: "a2b5a56fd7f9436d8e0e965881703a64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b5a56fd7f9436d8e0e965881703a64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34061 } } }
I20260924 07:22:22.299839 24562 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64 [sys.catalog]: This master's current role is: LEADER
W20260924 07:22:22.301199 24578 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:22:22.301245 24578 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:22:22.301313 24567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:22.301508 24567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:22.302419 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:22.300060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55903 (local address 127.0.0.1:34061)
0924 07:22:22.300267 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:22.300277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:22.300311 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:22:22.300432 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:22.300439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:22.300469 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:22.300890 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:22.301837 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:22.301841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:22.301841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:22.301845 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:22.301894 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:22.301910 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:22.302161 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:22.302191 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 07:22:22.302242 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260924 07:22:22.303229 24567 catalog_manager.cc:1347] Generated new cluster ID: 03993cbcd5a94719923df937dd271f96
I20260924 07:22:22.303283 24567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:22.303540 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:22.306335 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:22.305168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46075 (local address 127.0.0.1:34061)
0924 07:22:22.305208 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:22.305210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:22.305300 (+ 90us) server_negotiation.cc:407] Connection header received
0924 07:22:22.305392 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:22.305393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:22.305399 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:22.305427 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:22.306082 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:22.306083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:22.306083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:22.306088 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:22.306102 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:22.306106 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:22.306136 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:22.306253 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 07:22:22.306268 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:22:22.311233 24567 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:22.311403 24567 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:22.314051 24567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2b5a56fd7f9436d8e0e965881703a64: Generated new TSK 0
I20260924 07:22:22.314296 24567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:22.323199 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 24490
2026-09-24T07:22:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:22 dist-test-slave-rc1w krb5kdc[24589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:22 dist-test-slave-rc1w krb5kdc[24589](info): set up 2 sockets
Sep 24 07:22:22 dist-test-slave-rc1w krb5kdc[24589](info): commencing operation
krb5kdc: starting...
W20260924 07:22:23.365958 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:23 dist-test-slave-rc1w krb5kdc[24589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:22:23.485086 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:23.564007 24602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:23.564208 24602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:23.564231 24602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:23.565378 24602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:23.565420 24602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:23.565434 24602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:23.565446 24602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:23.565459 24602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:23.566927 24602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:23.567142 24602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:23.567361 24602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:23.573275 24608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:23.573246 24607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:23.573288 24610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:23.573704 24602 server_base.cc:1034] running on GCE node
I20260924 07:22:23.573922 24602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:23.574194 24602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:23.575358 24602 hybrid_clock.cc:648] HybridClock initialized: now 1790234543575348 us; error 38 us; skew 500 ppm
Sep 24 07:22:23 dist-test-slave-rc1w krb5kdc[24589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:23.577847 24602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:22:23.578454 24602 webserver.cc:460] Webserver started at http://127.0.0.1:33039/ using document root <none> and password file <none>
I20260924 07:22:23.578714 24602 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:23.578790 24602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:23.578950 24602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:23.580011 24602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "69cb247e162b4230a482ab076d7bb871"
format_stamp: "Formatted at 2026-09-24 07:22:23 on dist-test-slave-rc1w"
server_key: "64d7a2198dd93ce8f2f11bed95b49ea6"
server_key_iv: "01e2e44bf3b7802b2aaea520a7059fdd"
server_key_version: "encryptionkey@0"
I20260924 07:22:23.580394 24602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "69cb247e162b4230a482ab076d7bb871"
format_stamp: "Formatted at 2026-09-24 07:22:23 on dist-test-slave-rc1w"
server_key: "64d7a2198dd93ce8f2f11bed95b49ea6"
server_key_iv: "01e2e44bf3b7802b2aaea520a7059fdd"
server_key_version: "encryptionkey@0"
I20260924 07:22:23.581806 24602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:22:23.582573 24617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:23.582753 24602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:23.582851 24602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "69cb247e162b4230a482ab076d7bb871"
format_stamp: "Formatted at 2026-09-24 07:22:23 on dist-test-slave-rc1w"
server_key: "64d7a2198dd93ce8f2f11bed95b49ea6"
server_key_iv: "01e2e44bf3b7802b2aaea520a7059fdd"
server_key_version: "encryptionkey@0"
I20260924 07:22:23.582939 24602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:23.596465 24602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:23.598603 24602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:23.598762 24602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:23.602670 24602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38683
I20260924 07:22:23.602728 24669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38683 every 8 connection(s)
I20260924 07:22:23.603081 24602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:23.604101 24670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:23.606411 24670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871: Bootstrap starting.
I20260924 07:22:23.606990 24670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:23.607277 24670 log.cc:824] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:23.607939 24670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871: No bootstrap required, opened a new log
I20260924 07:22:23.609277 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 24602
I20260924 07:22:23.609416 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:23.609613 4057 external_mini_cluster.cc:1428] Setting key 4efd8833a7f316c2d8db31c7bf9eb48c
I20260924 07:22:23.609587 24670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69cb247e162b4230a482ab076d7bb871" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38683 } }
I20260924 07:22:23.609800 24670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:23.609853 24670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69cb247e162b4230a482ab076d7bb871, State: Initialized, Role: FOLLOWER
I20260924 07:22:23.610009 24670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69cb247e162b4230a482ab076d7bb871" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38683 } }
I20260924 07:22:23.610126 24670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:23.610213 24670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:23.610309 24670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:23.611166 24670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69cb247e162b4230a482ab076d7bb871" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38683 } }
I20260924 07:22:23.611284 24670 leader_election.cc:302] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69cb247e162b4230a482ab076d7bb871; no voters:
I20260924 07:22:23.611442 24670 leader_election.cc:288] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:23.611492 24675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:23.611616 24675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [term 1 LEADER]: Becoming Leader. State: Replica: 69cb247e162b4230a482ab076d7bb871, State: Running, Role: LEADER
I20260924 07:22:23.611701 24670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:23.611790 24675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69cb247e162b4230a482ab076d7bb871" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38683 } }
I20260924 07:22:23.612200 24675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69cb247e162b4230a482ab076d7bb871. Latest consensus state: current_term: 1 leader_uuid: "69cb247e162b4230a482ab076d7bb871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69cb247e162b4230a482ab076d7bb871" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38683 } } }
I20260924 07:22:23.612291 24677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69cb247e162b4230a482ab076d7bb871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69cb247e162b4230a482ab076d7bb871" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38683 } } }
I20260924 07:22:23.612336 24675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:23.612370 24677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:23.612550 24684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:23.613102 24684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:23.614611 24684 catalog_manager.cc:1347] Generated new cluster ID: df4166746147449b8258251e30adbdaf
I20260924 07:22:23.614663 24684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:22:23 dist-test-slave-rc1w krb5kdc[24589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:22:23.620930 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:23.610137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59949 (local address 127.0.0.1:38683)
0924 07:22:23.610309 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:23.610320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:23.610368 (+ 48us) server_negotiation.cc:407] Connection header received
0924 07:22:23.610495 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:23.610501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:23.610542 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:23.610674 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:23.611311 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:23.612631 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:23.613765 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:23.614326 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:23.614432 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:23.617318 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:23.617321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:23.617322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:23.617333 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:23.619246 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:23.619649 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:23.619652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:23.619652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:23.619761 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:23.619969 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:23.619969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:23.619970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:23.620260 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:23.620395 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:23.620580 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 07:22:23.620713 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260924 07:22:23.621915 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:23.630003 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:23.623284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44841 (local address 127.0.0.1:38683)
0924 07:22:23.623320 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:23.623321 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:23.623461 (+ 140us) server_negotiation.cc:407] Connection header received
0924 07:22:23.623539 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:23.623539 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:23.623557 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:23.623582 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:23.624178 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:23.624881 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:23.625858 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:23.626143 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:23.626177 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:23.627453 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:23.627454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:23.627455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:23.627461 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:23.628922 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:23.629247 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:23.629248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:23.629248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:23.629290 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:23.629465 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:23.629466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:23.629466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:23.629566 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:23.629645 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:23.629856 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 07:22:23.629923 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:22:23.639005 24684 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:23.639683 24684 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:23.645340 24684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69cb247e162b4230a482ab076d7bb871: Generated new TSK 0
I20260924 07:22:23.645529 24684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:23.652467 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 24602
2026-09-24T07:22:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:23 dist-test-slave-rc1w krb5kdc[24701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:23 dist-test-slave-rc1w krb5kdc[24701](info): set up 2 sockets
Sep 24 07:22:23 dist-test-slave-rc1w krb5kdc[24701](info): commencing operation
krb5kdc: starting...
W20260924 07:22:24.678090 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:24 dist-test-slave-rc1w krb5kdc[24701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:22:24.793397 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:24.872411 24714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:24.872635 24714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:24.872659 24714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:24.873828 24714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:24.873869 24714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:24.873883 24714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:24.873896 24714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:24.873947 24714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:24.875376 24714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:24.875618 24714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:24.875875 24714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:24.883602 24719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:24.883801 24722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:24.884016 24720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:24.884514 24714 server_base.cc:1034] running on GCE node
I20260924 07:22:24.884768 24714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:24.885062 24714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:24.886207 24714 hybrid_clock.cc:648] HybridClock initialized: now 1790234544886193 us; error 35 us; skew 500 ppm
Sep 24 07:22:24 dist-test-slave-rc1w krb5kdc[24701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:24.888532 24714 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:22:24.888962 24714 webserver.cc:460] Webserver started at http://127.0.0.1:38379/ using document root <none> and password file <none>
I20260924 07:22:24.889186 24714 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:24.889254 24714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:24.889405 24714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:24.890527 24714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "e687063051c24deab9dc256c32aed5f4"
format_stamp: "Formatted at 2026-09-24 07:22:24 on dist-test-slave-rc1w"
server_key: "a6ebfec42bc574e3f26251b3f0efbd76"
server_key_iv: "1851b42506399d45326672f4eef3deb6"
server_key_version: "encryptionkey@0"
I20260924 07:22:24.890899 24714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "e687063051c24deab9dc256c32aed5f4"
format_stamp: "Formatted at 2026-09-24 07:22:24 on dist-test-slave-rc1w"
server_key: "a6ebfec42bc574e3f26251b3f0efbd76"
server_key_iv: "1851b42506399d45326672f4eef3deb6"
server_key_version: "encryptionkey@0"
I20260924 07:22:24.892252 24714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:22:24.892962 24729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:24.893162 24714 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:22:24.893260 24714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "e687063051c24deab9dc256c32aed5f4"
format_stamp: "Formatted at 2026-09-24 07:22:24 on dist-test-slave-rc1w"
server_key: "a6ebfec42bc574e3f26251b3f0efbd76"
server_key_iv: "1851b42506399d45326672f4eef3deb6"
server_key_version: "encryptionkey@0"
I20260924 07:22:24.893342 24714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:24.909396 24714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:24.911290 24714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:24.911439 24714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:24.915248 24714 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44711
I20260924 07:22:24.915282 24781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44711 every 8 connection(s)
I20260924 07:22:24.915623 24714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:24.916636 24782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:24.918941 24782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4: Bootstrap starting.
I20260924 07:22:24.919538 24782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:24.919829 24782 log.cc:824] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:24.920523 24782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4: No bootstrap required, opened a new log
I20260924 07:22:24.921573 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 24714
I20260924 07:22:24.921720 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:24.921942 4057 external_mini_cluster.cc:1428] Setting key 8cc1d4ee01ef5ec9d8487b99dac5975c
I20260924 07:22:24.922456 24782 raft_consensus.cc:348] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e687063051c24deab9dc256c32aed5f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } }
I20260924 07:22:24.922647 24782 raft_consensus.cc:374] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:24.922714 24782 raft_consensus.cc:729] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e687063051c24deab9dc256c32aed5f4, State: Initialized, Role: FOLLOWER
I20260924 07:22:24.922848 24782 consensus_queue.cc:260] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e687063051c24deab9dc256c32aed5f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } }
I20260924 07:22:24.922948 24782 raft_consensus.cc:388] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:24.922991 24782 raft_consensus.cc:482] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:24.923041 24782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:24.923885 24782 raft_consensus.cc:504] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e687063051c24deab9dc256c32aed5f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } }
I20260924 07:22:24.924006 24782 leader_election.cc:302] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e687063051c24deab9dc256c32aed5f4; no voters:
I20260924 07:22:24.924162 24782 leader_election.cc:288] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:24.924199 24787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:24.924302 24787 raft_consensus.cc:686] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [term 1 LEADER]: Becoming Leader. State: Replica: e687063051c24deab9dc256c32aed5f4, State: Running, Role: LEADER
I20260924 07:22:24.924428 24787 consensus_queue.cc:237] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e687063051c24deab9dc256c32aed5f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } }
I20260924 07:22:24.924422 24782 sys_catalog.cc:564] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:24.924837 24788 sys_catalog.cc:455] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e687063051c24deab9dc256c32aed5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e687063051c24deab9dc256c32aed5f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } } }
I20260924 07:22:24.925056 24788 sys_catalog.cc:458] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:24.924999 24789 sys_catalog.cc:455] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e687063051c24deab9dc256c32aed5f4. Latest consensus state: current_term: 1 leader_uuid: "e687063051c24deab9dc256c32aed5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e687063051c24deab9dc256c32aed5f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } } }
I20260924 07:22:24.925302 24789 sys_catalog.cc:458] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:24.925484 24791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:24.926014 24791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:24.927841 24791 catalog_manager.cc:1347] Generated new cluster ID: 1684bf134e5d4ee09c7cb371349fa1df
I20260924 07:22:24.927891 24791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:24.928068 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:24.922523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45651 (local address 127.0.0.1:44711)
0924 07:22:24.922723 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:24.922733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:24.922760 (+ 27us) server_negotiation.cc:407] Connection header received
0924 07:22:24.922885 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:24.922891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:24.922933 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:24.923087 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:24.923293 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:24.924343 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:24.925476 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:24.926100 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:24.926208 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:24.927172 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:24.927177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:24.927177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:24.927398 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:24.927434 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:24.927448 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:24.927522 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:24.927748 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 07:22:24.927815 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260924 07:22:24.928897 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:24.937260 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:24.933977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47887 (local address 127.0.0.1:44711)
0924 07:22:24.934019 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:24.934021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:24.934103 (+ 82us) server_negotiation.cc:407] Connection header received
0924 07:22:24.934155 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:24.934156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:24.934166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:24.934234 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:24.934337 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:24.935039 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:24.936007 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:24.936307 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:24.936380 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:24.937021 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:24.937023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:24.937023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:24.937027 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:24.937043 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:24.937048 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:24.937105 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:24.937161 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 07:22:24.937184 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:22:24.947393 24791 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:24.948175 24791 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:24.956424 24791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e687063051c24deab9dc256c32aed5f4: Generated new TSK 0
I20260924 07:22:24.956627 24791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:24.963383 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 24714
2026-09-24T07:22:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:25 dist-test-slave-rc1w krb5kdc[24813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:25 dist-test-slave-rc1w krb5kdc[24813](info): set up 2 sockets
Sep 24 07:22:25 dist-test-slave-rc1w krb5kdc[24813](info): commencing operation
krb5kdc: starting...
W20260924 07:22:25.989886 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:26 dist-test-slave-rc1w krb5kdc[24813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:22:26.100847 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41507
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:26.180795 24826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:26.180992 24826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:26.181016 24826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:26.182090 24826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:26.182133 24826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:26.182147 24826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:26.182168 24826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:26.182179 24826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:26.183552 24826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:26.183729 24826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:26.183894 24826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:26.188531 24834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:26.188570 24831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:26.188545 24832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:26.188758 24826 server_base.cc:1034] running on GCE node
I20260924 07:22:26.189047 24826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:26.189291 24826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:26.190445 24826 hybrid_clock.cc:648] HybridClock initialized: now 1790234546190434 us; error 27 us; skew 500 ppm
Sep 24 07:22:26 dist-test-slave-rc1w krb5kdc[24813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:26.201229 24826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:22:26.201717 24826 webserver.cc:460] Webserver started at http://127.0.0.1:45845/ using document root <none> and password file <none>
I20260924 07:22:26.201928 24826 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:26.201968 24826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:26.202055 24826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:26.203038 24826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "f01f826cf8144ab09797dd51dee34837"
format_stamp: "Formatted at 2026-09-24 07:22:26 on dist-test-slave-rc1w"
server_key: "eb39741cdcde5d66234c6d8f47573b38"
server_key_iv: "0cdae8ad6fc4f22e63690ffe6ecf8702"
server_key_version: "encryptionkey@0"
I20260924 07:22:26.203343 24826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "f01f826cf8144ab09797dd51dee34837"
format_stamp: "Formatted at 2026-09-24 07:22:26 on dist-test-slave-rc1w"
server_key: "eb39741cdcde5d66234c6d8f47573b38"
server_key_iv: "0cdae8ad6fc4f22e63690ffe6ecf8702"
server_key_version: "encryptionkey@0"
I20260924 07:22:26.204552 24826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:22:26.205235 24841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:26.205405 24826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:26.205492 24826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "f01f826cf8144ab09797dd51dee34837"
format_stamp: "Formatted at 2026-09-24 07:22:26 on dist-test-slave-rc1w"
server_key: "eb39741cdcde5d66234c6d8f47573b38"
server_key_iv: "0cdae8ad6fc4f22e63690ffe6ecf8702"
server_key_version: "encryptionkey@0"
I20260924 07:22:26.205545 24826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:26.208741 24826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:26.210551 24826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:26.210654 24826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:26.214496 24826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41507
I20260924 07:22:26.214530 24893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41507 every 8 connection(s)
I20260924 07:22:26.214843 24826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:26.215353 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 24826
I20260924 07:22:26.215472 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:26.215649 4057 external_mini_cluster.cc:1428] Setting key c1135e36f6f4774c096647a56d7d1112
I20260924 07:22:26.215865 24894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:26.218250 24897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:26.216181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40023 (local address 127.0.0.1:41507)
0924 07:22:26.216383 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:26.216392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:26.216422 (+ 30us) server_negotiation.cc:407] Connection header received
0924 07:22:26.216990 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:26.216998 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:26.217032 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:26.217180 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:26.217826 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:26.217830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:26.217830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:26.217834 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:26.217883 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:26.217898 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:26.217975 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:26.218009 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 07:22:26.218044 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260924 07:22:26.219234 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:26.219365 24894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837: Bootstrap starting.
I20260924 07:22:26.220085 24894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:26.220432 24894 log.cc:824] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:26.221212 24894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837: No bootstrap required, opened a new log
I20260924 07:22:26.221868 24897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:26.220815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59769 (local address 127.0.0.1:41507)
0924 07:22:26.220857 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:26.220859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:26.220984 (+ 125us) server_negotiation.cc:407] Connection header received
0924 07:22:26.221051 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:26.221051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:26.221055 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:26.221098 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:26.221588 (+ 490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:26.221589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:26.221589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:26.221592 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:26.221604 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:26.221607 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:26.221644 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:26.221766 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 07:22:26.221776 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:22:26.223173 24894 raft_consensus.cc:348] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01f826cf8144ab09797dd51dee34837" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41507 } }
I20260924 07:22:26.223316 24894 raft_consensus.cc:374] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:26.223340 24894 raft_consensus.cc:729] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01f826cf8144ab09797dd51dee34837, State: Initialized, Role: FOLLOWER
I20260924 07:22:26.223471 24894 consensus_queue.cc:260] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01f826cf8144ab09797dd51dee34837" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41507 } }
I20260924 07:22:26.223546 24894 raft_consensus.cc:388] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:26.223585 24894 raft_consensus.cc:482] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:26.223634 24894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:26.224126 24894 raft_consensus.cc:504] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01f826cf8144ab09797dd51dee34837" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41507 } }
I20260924 07:22:26.224237 24894 leader_election.cc:302] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f01f826cf8144ab09797dd51dee34837; no voters:
I20260924 07:22:26.224424 24894 leader_election.cc:288] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:26.224454 24904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:26.224593 24904 raft_consensus.cc:686] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [term 1 LEADER]: Becoming Leader. State: Replica: f01f826cf8144ab09797dd51dee34837, State: Running, Role: LEADER
I20260924 07:22:26.224655 24894 sys_catalog.cc:564] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:26.224730 24904 consensus_queue.cc:237] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01f826cf8144ab09797dd51dee34837" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41507 } }
I20260924 07:22:26.225145 24906 sys_catalog.cc:455] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f01f826cf8144ab09797dd51dee34837. Latest consensus state: current_term: 1 leader_uuid: "f01f826cf8144ab09797dd51dee34837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01f826cf8144ab09797dd51dee34837" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41507 } } }
I20260924 07:22:26.225267 24906 sys_catalog.cc:458] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:26.225466 24913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:26.225133 24905 sys_catalog.cc:455] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f01f826cf8144ab09797dd51dee34837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01f826cf8144ab09797dd51dee34837" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41507 } } }
I20260924 07:22:26.225713 24905 sys_catalog.cc:458] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:26.225963 24913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:26.227380 24913 catalog_manager.cc:1347] Generated new cluster ID: 2b0a371c0d734b7c9800bf0c61c6c6b9
I20260924 07:22:26.227430 24913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:26.242817 24913 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:26.242981 24913 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:26.252127 24913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f01f826cf8144ab09797dd51dee34837: Generated new TSK 0
I20260924 07:22:26.252315 24913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:26.255734 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 24826
2026-09-24T07:22:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:26 dist-test-slave-rc1w krb5kdc[24925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:26 dist-test-slave-rc1w krb5kdc[24925](info): set up 2 sockets
Sep 24 07:22:26 dist-test-slave-rc1w krb5kdc[24925](info): commencing operation
krb5kdc: starting...
W20260924 07:22:27.285929 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:27 dist-test-slave-rc1w krb5kdc[24925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:22:27.396466 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:27.478542 24938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:27.478796 24938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:27.478833 24938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:27.480029 24938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:27.480099 24938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:27.480123 24938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:27.480151 24938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:27.480202 24938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:27.481757 24938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:27.482019 24938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:27.482255 24938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:27.486935 24943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:27.486989 24946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:27.487015 24944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:27.487279 24938 server_base.cc:1034] running on GCE node
I20260924 07:22:27.487509 24938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:27.487787 24938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:27.488974 24938 hybrid_clock.cc:648] HybridClock initialized: now 1790234547488965 us; error 37 us; skew 500 ppm
Sep 24 07:22:27 dist-test-slave-rc1w krb5kdc[24925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:27.491422 24938 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:22:27.491853 24938 webserver.cc:460] Webserver started at http://127.0.0.1:38907/ using document root <none> and password file <none>
I20260924 07:22:27.492075 24938 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:27.492138 24938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:27.492286 24938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:27.493343 24938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "9cc5c92cc38f40598f43e85408dfce43"
format_stamp: "Formatted at 2026-09-24 07:22:27 on dist-test-slave-rc1w"
server_key: "daf3400dcab65ac284e188ebe4028cbb"
server_key_iv: "8215950223b048d0081a9293eab65465"
server_key_version: "encryptionkey@0"
I20260924 07:22:27.493768 24938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "9cc5c92cc38f40598f43e85408dfce43"
format_stamp: "Formatted at 2026-09-24 07:22:27 on dist-test-slave-rc1w"
server_key: "daf3400dcab65ac284e188ebe4028cbb"
server_key_iv: "8215950223b048d0081a9293eab65465"
server_key_version: "encryptionkey@0"
I20260924 07:22:27.495121 24938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:22:27.495872 24953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:27.496039 24938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:22:27.496132 24938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "9cc5c92cc38f40598f43e85408dfce43"
format_stamp: "Formatted at 2026-09-24 07:22:27 on dist-test-slave-rc1w"
server_key: "daf3400dcab65ac284e188ebe4028cbb"
server_key_iv: "8215950223b048d0081a9293eab65465"
server_key_version: "encryptionkey@0"
I20260924 07:22:27.496227 24938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:27.504755 24938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:27.506539 24938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:27.506688 24938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:27.510823 24938 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38921
I20260924 07:22:27.510854 25005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38921 every 8 connection(s)
I20260924 07:22:27.511183 24938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:27.512084 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:27.514362 25006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43: Bootstrap starting.
I20260924 07:22:27.514968 25006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:27.515254 25006 log.cc:824] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:27.515930 25006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43: No bootstrap required, opened a new log
I20260924 07:22:27.517459 25006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc5c92cc38f40598f43e85408dfce43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38921 } }
I20260924 07:22:27.517617 25006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:27.517664 25006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc5c92cc38f40598f43e85408dfce43, State: Initialized, Role: FOLLOWER
I20260924 07:22:27.517831 25006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc5c92cc38f40598f43e85408dfce43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38921 } }
I20260924 07:22:27.517910 25006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:27.517954 25006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:27.518004 25006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:27.518501 25006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc5c92cc38f40598f43e85408dfce43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38921 } }
I20260924 07:22:27.518615 25006 leader_election.cc:302] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cc5c92cc38f40598f43e85408dfce43; no voters:
I20260924 07:22:27.518790 25006 leader_election.cc:288] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:27.518833 25009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:27.519013 25009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [term 1 LEADER]: Becoming Leader. State: Replica: 9cc5c92cc38f40598f43e85408dfce43, State: Running, Role: LEADER
I20260924 07:22:27.519033 25006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:27.519104 25009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc5c92cc38f40598f43e85408dfce43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38921 } }
I20260924 07:22:27.519462 25010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cc5c92cc38f40598f43e85408dfce43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc5c92cc38f40598f43e85408dfce43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38921 } } }
I20260924 07:22:27.519466 25011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc5c92cc38f40598f43e85408dfce43. Latest consensus state: current_term: 1 leader_uuid: "9cc5c92cc38f40598f43e85408dfce43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc5c92cc38f40598f43e85408dfce43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38921 } } }
I20260924 07:22:27.519610 25010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:27.519673 25011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:27.520134 25013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:27.520635 25013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:27.520934 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 24938
I20260924 07:22:27.521054 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:27.521231 4057 external_mini_cluster.cc:1428] Setting key f0d96a27e09c70e8aecba2c1ce28a691
I20260924 07:22:27.522526 25013 catalog_manager.cc:1347] Generated new cluster ID: c6a31e23ec65462a9b7520c1ce743af7
I20260924 07:22:27.522575 25013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:27.527661 25013 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:27.528316 25013 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 07:22:27 dist-test-slave-rc1w krb5kdc[24925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:22:27.532178 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:27.521813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:45777 (local address 127.0.0.1:38921)
0924 07:22:27.521971 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:27.521980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:27.522042 (+ 62us) server_negotiation.cc:407] Connection header received
0924 07:22:27.522189 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:27.522195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:27.522240 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:27.522387 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:27.523139 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:27.524618 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:27.525621 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:27.526255 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:27.526366 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:27.528967 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:27.528971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:27.528971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:27.528995 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:27.530700 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:27.531076 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:27.531079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:27.531079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:27.531170 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:27.531413 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:27.531415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:27.531415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:27.531627 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:27.531808 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:27.531897 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 07:22:27.531996 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":94,"threads_started":1}
I20260924 07:22:27.533092 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:27.535713 25013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cc5c92cc38f40598f43e85408dfce43: Generated new TSK 0
I20260924 07:22:27.535882 25013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:27.542032 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:27.534470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:34025 (local address 127.0.0.1:38921)
0924 07:22:27.534507 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:27.534509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:27.535563 (+ 1054us) server_negotiation.cc:407] Connection header received
0924 07:22:27.535583 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:27.535584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:27.535593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:27.535640 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:27.536308 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:27.537106 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:27.537837 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:27.538154 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:27.538225 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:27.539348 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:27.539349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:27.539350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:27.539356 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:27.540839 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:27.541182 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:27.541184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:27.541184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:27.541243 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:27.541468 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:27.541469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:27.541470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:27.541604 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:27.541732 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:27.541878 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 07:22:27.541943 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:22:27.544032 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 24938
W20260924 07:22:27.548544 24923 connection.cc:535] client connection to 127.0.0.1:38921 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38921): Connection reset by peer (error 104)
2026-09-24T07:22:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:27 dist-test-slave-rc1w krb5kdc[25037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:27 dist-test-slave-rc1w krb5kdc[25037](info): set up 2 sockets
Sep 24 07:22:27 dist-test-slave-rc1w krb5kdc[25037](info): commencing operation
krb5kdc: starting...
W20260924 07:22:28.573797 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:28 dist-test-slave-rc1w krb5kdc[25037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:22:28.686553 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:28.765780 25050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:28.765980 25050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:28.766002 25050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:28.767139 25050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:28.767189 25050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:28.767204 25050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:28.767238 25050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:28.767266 25050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:28.768716 25050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:28.768952 25050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:28.769172 25050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:28.773976 25056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:28.773983 25055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:28.774014 25050 server_base.cc:1034] running on GCE node
W20260924 07:22:28.773996 25058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:28.774500 25050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:28.774776 25050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:28.775955 25050 hybrid_clock.cc:648] HybridClock initialized: now 1790234548775945 us; error 30 us; skew 500 ppm
Sep 24 07:22:28 dist-test-slave-rc1w krb5kdc[25037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:28.778338 25050 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:22:28.778764 25050 webserver.cc:460] Webserver started at http://127.0.0.1:38605/ using document root <none> and password file <none>
I20260924 07:22:28.778977 25050 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:28.779042 25050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:28.779187 25050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:28.780213 25050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "5d0420e64282450496b234cea5d5178d"
format_stamp: "Formatted at 2026-09-24 07:22:28 on dist-test-slave-rc1w"
server_key: "a503b0a30ac553409d503ba2356b5712"
server_key_iv: "9c2dd8734f0a50d8fbe702a1a59588f9"
server_key_version: "encryptionkey@0"
I20260924 07:22:28.780563 25050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "5d0420e64282450496b234cea5d5178d"
format_stamp: "Formatted at 2026-09-24 07:22:28 on dist-test-slave-rc1w"
server_key: "a503b0a30ac553409d503ba2356b5712"
server_key_iv: "9c2dd8734f0a50d8fbe702a1a59588f9"
server_key_version: "encryptionkey@0"
I20260924 07:22:28.781863 25050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:22:28.782526 25065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:28.782685 25050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:28.782794 25050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "5d0420e64282450496b234cea5d5178d"
format_stamp: "Formatted at 2026-09-24 07:22:28 on dist-test-slave-rc1w"
server_key: "a503b0a30ac553409d503ba2356b5712"
server_key_iv: "9c2dd8734f0a50d8fbe702a1a59588f9"
server_key_version: "encryptionkey@0"
I20260924 07:22:28.782863 25050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:28.804956 25050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:28.806887 25050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:28.807044 25050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:28.810925 25050 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37837
I20260924 07:22:28.810977 25117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37837 every 8 connection(s)
I20260924 07:22:28.811313 25050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:28.812259 25118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:28.814509 25118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d: Bootstrap starting.
I20260924 07:22:28.815066 25118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:28.815340 25118 log.cc:824] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:28.815992 25118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d: No bootstrap required, opened a new log
I20260924 07:22:28.817512 25118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0420e64282450496b234cea5d5178d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37837 } }
I20260924 07:22:28.817642 25118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:28.817719 25118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d0420e64282450496b234cea5d5178d, State: Initialized, Role: FOLLOWER
I20260924 07:22:28.817875 25118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0420e64282450496b234cea5d5178d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37837 } }
I20260924 07:22:28.817953 25118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:28.817999 25118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:28.818063 25118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:28.818568 25118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0420e64282450496b234cea5d5178d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37837 } }
I20260924 07:22:28.818675 25118 leader_election.cc:302] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d0420e64282450496b234cea5d5178d; no voters:
I20260924 07:22:28.818883 25118 leader_election.cc:288] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:28.818939 25121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:28.819097 25121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [term 1 LEADER]: Becoming Leader. State: Replica: 5d0420e64282450496b234cea5d5178d, State: Running, Role: LEADER
I20260924 07:22:28.819123 25118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:28.819181 25121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0420e64282450496b234cea5d5178d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37837 } }
I20260924 07:22:28.819553 25122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d0420e64282450496b234cea5d5178d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0420e64282450496b234cea5d5178d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37837 } } }
I20260924 07:22:28.819657 25122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:28.819554 25123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d0420e64282450496b234cea5d5178d. Latest consensus state: current_term: 1 leader_uuid: "5d0420e64282450496b234cea5d5178d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0420e64282450496b234cea5d5178d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37837 } } }
I20260924 07:22:28.819891 25123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:28.820127 25125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:28.820616 25125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:28.820989 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 25050
I20260924 07:22:28.821099 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:28.821313 4057 external_mini_cluster.cc:1428] Setting key 8f299a8920ef796ab77a11881f417d38
I20260924 07:22:28.822379 25125 catalog_manager.cc:1347] Generated new cluster ID: b5930f2119e441ad828e44e634eff281
I20260924 07:22:28.822434 25125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:28.826536 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:28.821725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:39821 (local address 127.0.0.1:37837)
0924 07:22:28.822414 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:28.822420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:28.822463 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:22:28.822574 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:28.822579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:28.822621 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:28.822771 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:28.822968 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:28.823882 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:28.824688 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:28.825120 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:28.825234 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:28.825952 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:28.825956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:28.825956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:28.826130 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:28.826162 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:28.826173 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:28.826262 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:28.826342 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 07:22:28.826369 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":83,"threads_started":1}
I20260924 07:22:28.827519 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:28.832404 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:28.828812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:54767 (local address 127.0.0.1:37837)
0924 07:22:28.828853 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:28.828855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:28.829823 (+ 968us) server_negotiation.cc:407] Connection header received
0924 07:22:28.829897 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:28.829898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:28.829907 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:28.829946 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:28.830091 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:28.830568 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:28.831264 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:28.831472 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:28.831544 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:28.832161 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:28.832162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:28.832163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:28.832167 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:28.832181 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:28.832185 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:28.832245 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:28.832303 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 07:22:28.832314 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:22:28.832962 25125 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:28.833609 25125 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:28.839278 25125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d0420e64282450496b234cea5d5178d: Generated new TSK 0
I20260924 07:22:28.839466 25125 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:28.842423 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 25050
W20260924 07:22:28.846024 25035 connection.cc:535] client connection to 127.0.0.1:37837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37837): Connection reset by peer (error 104)
2026-09-24T07:22:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:28 dist-test-slave-rc1w krb5kdc[25149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:28 dist-test-slave-rc1w krb5kdc[25149](info): set up 2 sockets
Sep 24 07:22:28 dist-test-slave-rc1w krb5kdc[25149](info): commencing operation
krb5kdc: starting...
W20260924 07:22:29.869861 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:29 dist-test-slave-rc1w krb5kdc[25149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:22:29.988575 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:30.068109 25162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:30.068307 25162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:30.068331 25162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:30.069458 25162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:30.069500 25162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:30.069514 25162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:30.069527 25162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:30.069540 25162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:30.071022 25162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:30.071264 25162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:30.071475 25162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:30.076282 25167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:30.076287 25168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:30.076282 25170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:30.076386 25162 server_base.cc:1034] running on GCE node
I20260924 07:22:30.076735 25162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:30.077018 25162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:30.078172 25162 hybrid_clock.cc:648] HybridClock initialized: now 1790234550078158 us; error 40 us; skew 500 ppm
Sep 24 07:22:30 dist-test-slave-rc1w krb5kdc[25149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:30.080587 25162 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:22:30.080991 25162 webserver.cc:460] Webserver started at http://127.0.0.1:37285/ using document root <none> and password file <none>
I20260924 07:22:30.081204 25162 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:30.081280 25162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:30.081422 25162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:30.082508 25162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "bfd6efaa10b445d8b18eb05e58400a74"
format_stamp: "Formatted at 2026-09-24 07:22:30 on dist-test-slave-rc1w"
server_key: "1c646f8602882365a8223c278cc475b4"
server_key_iv: "fc9bc6af21260b6d3acfe9500c3d2765"
server_key_version: "encryptionkey@0"
I20260924 07:22:30.082862 25162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "bfd6efaa10b445d8b18eb05e58400a74"
format_stamp: "Formatted at 2026-09-24 07:22:30 on dist-test-slave-rc1w"
server_key: "1c646f8602882365a8223c278cc475b4"
server_key_iv: "fc9bc6af21260b6d3acfe9500c3d2765"
server_key_version: "encryptionkey@0"
I20260924 07:22:30.084133 25162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:22:30.084820 25177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:30.084978 25162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:22:30.085071 25162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "bfd6efaa10b445d8b18eb05e58400a74"
format_stamp: "Formatted at 2026-09-24 07:22:30 on dist-test-slave-rc1w"
server_key: "1c646f8602882365a8223c278cc475b4"
server_key_iv: "fc9bc6af21260b6d3acfe9500c3d2765"
server_key_version: "encryptionkey@0"
I20260924 07:22:30.085155 25162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:30.088397 25162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:30.090657 25162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:30.090811 25162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:30.094635 25162 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43247
I20260924 07:22:30.094671 25229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43247 every 8 connection(s)
I20260924 07:22:30.094975 25162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:30.095872 25230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:30.098161 25230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74: Bootstrap starting.
I20260924 07:22:30.098719 25230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:30.098992 25230 log.cc:824] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:30.099643 25230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74: No bootstrap required, opened a new log
I20260924 07:22:30.101146 25230 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd6efaa10b445d8b18eb05e58400a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43247 } }
I20260924 07:22:30.101276 25230 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:30.101333 25230 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd6efaa10b445d8b18eb05e58400a74, State: Initialized, Role: FOLLOWER
I20260924 07:22:30.101495 25230 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd6efaa10b445d8b18eb05e58400a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43247 } }
I20260924 07:22:30.101557 25230 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:30.101578 25230 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:30.101639 25230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:30.102164 25230 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd6efaa10b445d8b18eb05e58400a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43247 } }
I20260924 07:22:30.102272 25230 leader_election.cc:302] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfd6efaa10b445d8b18eb05e58400a74; no voters:
I20260924 07:22:30.102447 25230 leader_election.cc:288] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:30.102499 25233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:30.102650 25233 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [term 1 LEADER]: Becoming Leader. State: Replica: bfd6efaa10b445d8b18eb05e58400a74, State: Running, Role: LEADER
I20260924 07:22:30.102690 25230 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:30.102780 25233 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd6efaa10b445d8b18eb05e58400a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43247 } }
I20260924 07:22:30.103003 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 25162
I20260924 07:22:30.103118 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:30.103201 25235 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfd6efaa10b445d8b18eb05e58400a74. Latest consensus state: current_term: 1 leader_uuid: "bfd6efaa10b445d8b18eb05e58400a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd6efaa10b445d8b18eb05e58400a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43247 } } }
I20260924 07:22:30.103274 25235 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:30.103355 4057 external_mini_cluster.cc:1428] Setting key 364e45ac28a2094f8208160da6ee5f9e
I20260924 07:22:30.103405 25234 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfd6efaa10b445d8b18eb05e58400a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd6efaa10b445d8b18eb05e58400a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43247 } } }
I20260924 07:22:30.103459 25234 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:30.103741 25237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:30.104346 25237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:30.105875 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:30.103960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:52087 (local address 127.0.0.1:43247)
0924 07:22:30.104160 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:30.104170 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:30.104205 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:22:30.104334 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:30.104343 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:30.104376 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:30.104813 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:30.105300 (+ 487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:30.105303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:30.105304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:30.105308 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:30.105347 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:30.105362 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:30.105454 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:30.105617 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 07:22:30.105665 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260924 07:22:30.105903 25237 catalog_manager.cc:1347] Generated new cluster ID: 59003df593ac4332804a90235b01edea
I20260924 07:22:30.105938 25237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:30.110574 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:30.113179 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:30.112117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:42299 (local address 127.0.0.1:43247)
0924 07:22:30.112153 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:30.112154 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:30.112282 (+ 128us) server_negotiation.cc:407] Connection header received
0924 07:22:30.112349 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:30.112350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:30.112368 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:30.112421 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:30.112981 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:30.112982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:30.112982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:30.112986 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:30.113000 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:30.113004 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:30.113054 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:30.113098 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 07:22:30.113110 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:22:30.126626 25237 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:30.126817 25237 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:30.129755 25237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfd6efaa10b445d8b18eb05e58400a74: Generated new TSK 0
I20260924 07:22:30.129920 25237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:30.133508 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 25162
2026-09-24T07:22:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:30 dist-test-slave-rc1w krb5kdc[25261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:30 dist-test-slave-rc1w krb5kdc[25261](info): set up 2 sockets
Sep 24 07:22:30 dist-test-slave-rc1w krb5kdc[25261](info): commencing operation
krb5kdc: starting...
W20260924 07:22:31.165871 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:31 dist-test-slave-rc1w krb5kdc[25261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:22:31.279922 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:31.359966 25274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:31.360162 25274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:31.360185 25274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:31.361331 25274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:31.361377 25274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:31.361390 25274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:31.361403 25274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:31.361415 25274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:31.362905 25274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:31.363150 25274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:31.363375 25274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:31.368073 25279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:31.368170 25282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:31.368304 25274 server_base.cc:1034] running on GCE node
W20260924 07:22:31.368328 25280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:31.368631 25274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:31.368875 25274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:31.370101 25274 hybrid_clock.cc:648] HybridClock initialized: now 1790234551370071 us; error 44 us; skew 500 ppm
Sep 24 07:22:31 dist-test-slave-rc1w krb5kdc[25261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:31.384197 25274 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:22:31.384652 25274 webserver.cc:460] Webserver started at http://127.0.0.1:36087/ using document root <none> and password file <none>
I20260924 07:22:31.384881 25274 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:31.384984 25274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:31.385174 25274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:31.386374 25274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "41a9eac453574c7d9e826bf4f887d091"
format_stamp: "Formatted at 2026-09-24 07:22:31 on dist-test-slave-rc1w"
server_key: "cf62b47811293b0d395cc5e190b582ea"
server_key_iv: "bd3cffd4598f610f72285ab02e3a3768"
server_key_version: "encryptionkey@0"
I20260924 07:22:31.386732 25274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "41a9eac453574c7d9e826bf4f887d091"
format_stamp: "Formatted at 2026-09-24 07:22:31 on dist-test-slave-rc1w"
server_key: "cf62b47811293b0d395cc5e190b582ea"
server_key_iv: "bd3cffd4598f610f72285ab02e3a3768"
server_key_version: "encryptionkey@0"
I20260924 07:22:31.388065 25274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:22:31.388751 25289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:31.388932 25274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:31.389019 25274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "41a9eac453574c7d9e826bf4f887d091"
format_stamp: "Formatted at 2026-09-24 07:22:31 on dist-test-slave-rc1w"
server_key: "cf62b47811293b0d395cc5e190b582ea"
server_key_iv: "bd3cffd4598f610f72285ab02e3a3768"
server_key_version: "encryptionkey@0"
I20260924 07:22:31.389072 25274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:31.402086 25274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:31.403806 25274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:31.403911 25274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:31.407682 25274 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33955
I20260924 07:22:31.407737 25341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33955 every 8 connection(s)
I20260924 07:22:31.408059 25274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:31.409025 25342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:31.411325 25342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091: Bootstrap starting.
I20260924 07:22:31.411895 25342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:31.412175 25342 log.cc:824] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:31.412848 25342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091: No bootstrap required, opened a new log
I20260924 07:22:31.414433 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 25274
I20260924 07:22:31.414569 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:31.414568 25342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a9eac453574c7d9e826bf4f887d091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33955 } }
I20260924 07:22:31.414758 25342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:31.414789 4057 external_mini_cluster.cc:1428] Setting key e5489e523b0311271376efcbba9fa8c0
I20260924 07:22:31.414808 25342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a9eac453574c7d9e826bf4f887d091, State: Initialized, Role: FOLLOWER
I20260924 07:22:31.414973 25342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a9eac453574c7d9e826bf4f887d091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33955 } }
I20260924 07:22:31.415055 25342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:31.415092 25342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:31.415143 25342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:31.416043 25342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a9eac453574c7d9e826bf4f887d091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33955 } }
I20260924 07:22:31.416133 25342 leader_election.cc:302] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41a9eac453574c7d9e826bf4f887d091; no voters:
I20260924 07:22:31.416316 25342 leader_election.cc:288] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:31.416342 25347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:31.416587 25347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [term 1 LEADER]: Becoming Leader. State: Replica: 41a9eac453574c7d9e826bf4f887d091, State: Running, Role: LEADER
I20260924 07:22:31.416725 25342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:31.416734 25347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a9eac453574c7d9e826bf4f887d091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33955 } }
I20260924 07:22:31.417160 25349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41a9eac453574c7d9e826bf4f887d091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a9eac453574c7d9e826bf4f887d091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33955 } } }
I20260924 07:22:31.417269 25349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:31.417182 25348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41a9eac453574c7d9e826bf4f887d091. Latest consensus state: current_term: 1 leader_uuid: "41a9eac453574c7d9e826bf4f887d091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a9eac453574c7d9e826bf4f887d091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33955 } } }
I20260924 07:22:31.417409 25348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:31.417788 25355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:31.418309 25355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:31.419904 25355 catalog_manager.cc:1347] Generated new cluster ID: 462a1a18586c42eeb85ca3433f7595cd
I20260924 07:22:31.419962 25355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:22:31 dist-test-slave-rc1w krb5kdc[25261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:22:31.426286 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:31.415293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:51019 (local address 127.0.0.1:33955)
0924 07:22:31.415481 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:31.415490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:31.415520 (+ 30us) server_negotiation.cc:407] Connection header received
0924 07:22:31.415619 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:31.415624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:31.415668 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:31.415800 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:31.416619 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:31.417973 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:31.419035 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:31.419586 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:31.419686 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:31.422523 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:31.422527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:31.422527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:31.422538 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:31.424543 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:31.424964 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:31.424967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:31.424967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:31.425084 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:31.425328 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:31.425329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:31.425329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:31.425708 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:31.425886 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:31.425976 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 07:22:31.426075 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260924 07:22:31.427253 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:31.436116 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:31.428691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:45477 (local address 127.0.0.1:33955)
0924 07:22:31.428751 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:31.428753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:31.428922 (+ 169us) server_negotiation.cc:407] Connection header received
0924 07:22:31.428981 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:31.428982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:31.428991 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:31.429042 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:31.429742 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:31.430448 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:31.431533 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:31.431846 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:31.431903 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:31.433293 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:31.433294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:31.433294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:31.433300 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:31.434910 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:31.435328 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:31.435329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:31.435330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:31.435384 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:31.435601 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:31.435601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:31.435602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:31.435719 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:31.435820 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:31.435973 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 07:22:31.436030 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:22:31.444097 25355 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:31.444815 25355 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:31.451828 25355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41a9eac453574c7d9e826bf4f887d091: Generated new TSK 0
I20260924 07:22:31.452014 25355 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:31.454674 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 25274
W20260924 07:22:31.458971 25259 connection.cc:535] client connection to 127.0.0.1:33955 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33955): Connection reset by peer (error 104)
2026-09-24T07:22:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:31 dist-test-slave-rc1w krb5kdc[25373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:31 dist-test-slave-rc1w krb5kdc[25373](info): set up 2 sockets
Sep 24 07:22:31 dist-test-slave-rc1w krb5kdc[25373](info): commencing operation
krb5kdc: starting...
W20260924 07:22:32.489930 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:32 dist-test-slave-rc1w krb5kdc[25373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:22:32.608413 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:32.692003 25386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:32.692245 25386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:32.692283 25386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:32.693434 25386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:32.693497 25386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:32.693521 25386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:32.693549 25386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:32.693579 25386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:32.695056 25386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:32.695333 25386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:32.695545 25386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:32.700266 25391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:32.700266 25392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:32.700290 25394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:32.700660 25386 server_base.cc:1034] running on GCE node
I20260924 07:22:32.700905 25386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:32.701186 25386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:32.702348 25386 hybrid_clock.cc:648] HybridClock initialized: now 1790234552702328 us; error 40 us; skew 500 ppm
Sep 24 07:22:32 dist-test-slave-rc1w krb5kdc[25373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:32.704857 25386 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:22:32.705260 25386 webserver.cc:460] Webserver started at http://127.0.0.1:44409/ using document root <none> and password file <none>
I20260924 07:22:32.705456 25386 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:32.705504 25386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:32.705593 25386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:32.706646 25386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "4d976194d09b4303b777635379db35aa"
format_stamp: "Formatted at 2026-09-24 07:22:32 on dist-test-slave-rc1w"
server_key: "6807b9c174d030cf1a484754abd9d363"
server_key_iv: "78dc0b743dfa403fb46caa3dde4da5e0"
server_key_version: "encryptionkey@0"
I20260924 07:22:32.706936 25386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "4d976194d09b4303b777635379db35aa"
format_stamp: "Formatted at 2026-09-24 07:22:32 on dist-test-slave-rc1w"
server_key: "6807b9c174d030cf1a484754abd9d363"
server_key_iv: "78dc0b743dfa403fb46caa3dde4da5e0"
server_key_version: "encryptionkey@0"
I20260924 07:22:32.708184 25386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:22:32.708844 25401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:32.709029 25386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:32.709110 25386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "4d976194d09b4303b777635379db35aa"
format_stamp: "Formatted at 2026-09-24 07:22:32 on dist-test-slave-rc1w"
server_key: "6807b9c174d030cf1a484754abd9d363"
server_key_iv: "78dc0b743dfa403fb46caa3dde4da5e0"
server_key_version: "encryptionkey@0"
I20260924 07:22:32.709165 25386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:32.726024 25386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:32.727840 25386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:32.727955 25386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:32.731789 25386 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38583
I20260924 07:22:32.731842 25453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38583 every 8 connection(s)
I20260924 07:22:32.732188 25386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:32.733086 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 25386
I20260924 07:22:32.733098 25454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:32.733178 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:32.733420 4057 external_mini_cluster.cc:1428] Setting key 422d93eb5efa1ae530626d7e81f3f949
I20260924 07:22:32.736569 25454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa: Bootstrap starting.
I20260924 07:22:32.737196 25454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:32.737591 25454 log.cc:824] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:32.738487 25454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa: No bootstrap required, opened a new log
I20260924 07:22:32.739437 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:32.734007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:42749 (local address 127.0.0.1:38583)
0924 07:22:32.734203 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:32.734213 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:32.734246 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:22:32.734741 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:32.734750 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:32.734817 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:32.734965 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:32.735120 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:32.736493 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:32.737304 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:32.737938 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:32.738066 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:32.738758 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:32.738762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:32.738763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:32.738960 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:32.739001 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:32.739016 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:32.739104 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:32.739194 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 07:22:32.739221 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260924 07:22:32.740513 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:32.740948 25454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d976194d09b4303b777635379db35aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38583 } }
I20260924 07:22:32.741135 25454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:32.741178 25454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d976194d09b4303b777635379db35aa, State: Initialized, Role: FOLLOWER
I20260924 07:22:32.741312 25454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d976194d09b4303b777635379db35aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38583 } }
I20260924 07:22:32.741391 25454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:32.741421 25454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:32.741467 25454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:32.742255 25454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d976194d09b4303b777635379db35aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38583 } }
I20260924 07:22:32.742373 25454 leader_election.cc:302] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d976194d09b4303b777635379db35aa; no voters:
I20260924 07:22:32.742554 25454 leader_election.cc:288] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:32.742584 25464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:32.742717 25464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [term 1 LEADER]: Becoming Leader. State: Replica: 4d976194d09b4303b777635379db35aa, State: Running, Role: LEADER
I20260924 07:22:32.742806 25454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:32.742871 25464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d976194d09b4303b777635379db35aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38583 } }
I20260924 07:22:32.743244 25464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d976194d09b4303b777635379db35aa. Latest consensus state: current_term: 1 leader_uuid: "4d976194d09b4303b777635379db35aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d976194d09b4303b777635379db35aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38583 } } }
I20260924 07:22:32.743482 25464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:32.743407 25466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d976194d09b4303b777635379db35aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d976194d09b4303b777635379db35aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38583 } } }
I20260924 07:22:32.743682 25473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:32.743707 25466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:32.744231 25473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:32.745479 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:32.742106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:46545 (local address 127.0.0.1:38583)
0924 07:22:32.742147 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:32.742149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:32.742244 (+ 95us) server_negotiation.cc:407] Connection header received
0924 07:22:32.742269 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:32.742270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:32.742281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:32.742345 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:32.742498 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:32.743131 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:32.744203 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:32.744475 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:32.744519 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:32.745261 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:32.745262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:32.745263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:32.745267 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:32.745279 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:32.745283 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:32.745337 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:32.745397 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 07:22:32.745406 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:22:32.750851 25473 catalog_manager.cc:1347] Generated new cluster ID: f61d129ae5774cbcb6fcd197750e3199
I20260924 07:22:32.750914 25473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:32.779629 25473 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:32.780177 25473 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:32.787235 25473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d976194d09b4303b777635379db35aa: Generated new TSK 0
I20260924 07:22:32.787431 25473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:32.800076 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 25386
W20260924 07:22:32.804554 25371 connection.cc:535] client connection to 127.0.0.1:38583 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38583): Connection reset by peer (error 104)
2026-09-24T07:22:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:32 dist-test-slave-rc1w krb5kdc[25485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:32 dist-test-slave-rc1w krb5kdc[25485](info): set up 2 sockets
Sep 24 07:22:32 dist-test-slave-rc1w krb5kdc[25485](info): commencing operation
krb5kdc: starting...
W20260924 07:22:33.834080 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:33 dist-test-slave-rc1w krb5kdc[25485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:22:33.948422 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:34.029564 25498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:34.029831 25498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:34.029860 25498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:34.031023 25498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:34.031061 25498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:34.031076 25498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:34.031121 25498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:34.031147 25498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:34.032632 25498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:34.032867 25498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:34.033109 25498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:34.037887 25506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:34.037879 25503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:34.037879 25504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:34.037959 25498 server_base.cc:1034] running on GCE node
I20260924 07:22:34.038340 25498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:34.038569 25498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:34.039731 25498 hybrid_clock.cc:648] HybridClock initialized: now 1790234554039709 us; error 32 us; skew 500 ppm
Sep 24 07:22:34 dist-test-slave-rc1w krb5kdc[25485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:34.042178 25498 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:22:34.042547 25498 webserver.cc:460] Webserver started at http://127.0.0.1:34155/ using document root <none> and password file <none>
I20260924 07:22:34.042766 25498 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:34.042833 25498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:34.042974 25498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:34.044051 25498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "8c8acc4e75974928898e2c939e7934e3"
format_stamp: "Formatted at 2026-09-24 07:22:34 on dist-test-slave-rc1w"
server_key: "8b273364bec84ce293e06b496cf378a3"
server_key_iv: "2f746db7b2d22ab281268930ae42fd14"
server_key_version: "encryptionkey@0"
I20260924 07:22:34.044409 25498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "8c8acc4e75974928898e2c939e7934e3"
format_stamp: "Formatted at 2026-09-24 07:22:34 on dist-test-slave-rc1w"
server_key: "8b273364bec84ce293e06b496cf378a3"
server_key_iv: "2f746db7b2d22ab281268930ae42fd14"
server_key_version: "encryptionkey@0"
I20260924 07:22:34.045768 25498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:22:34.046490 25513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:34.046656 25498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:34.046753 25498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "8c8acc4e75974928898e2c939e7934e3"
format_stamp: "Formatted at 2026-09-24 07:22:34 on dist-test-slave-rc1w"
server_key: "8b273364bec84ce293e06b496cf378a3"
server_key_iv: "2f746db7b2d22ab281268930ae42fd14"
server_key_version: "encryptionkey@0"
I20260924 07:22:34.046834 25498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:34.050021 25498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:34.052208 25498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:34.052356 25498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:34.056067 25498 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45911
I20260924 07:22:34.056113 25565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45911 every 8 connection(s)
I20260924 07:22:34.056434 25498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:34.057377 25566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:34.059552 25566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3: Bootstrap starting.
I20260924 07:22:34.060071 25566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:34.060300 25566 log.cc:824] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:34.060894 25566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3: No bootstrap required, opened a new log
I20260924 07:22:34.062481 25566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8acc4e75974928898e2c939e7934e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260924 07:22:34.062611 25566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:34.062633 25566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c8acc4e75974928898e2c939e7934e3, State: Initialized, Role: FOLLOWER
I20260924 07:22:34.062790 25566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8acc4e75974928898e2c939e7934e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260924 07:22:34.062860 25566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:34.062881 25566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:34.062945 25566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:34.062973 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 25498
I20260924 07:22:34.063100 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:34.063304 4057 external_mini_cluster.cc:1428] Setting key a10d194e94e266c8b9ca416346d95289
I20260924 07:22:34.063591 25566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8acc4e75974928898e2c939e7934e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260924 07:22:34.063709 25566 leader_election.cc:302] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c8acc4e75974928898e2c939e7934e3; no voters:
I20260924 07:22:34.063877 25566 leader_election.cc:288] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:34.063912 25570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:34.064128 25570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [term 1 LEADER]: Becoming Leader. State: Replica: 8c8acc4e75974928898e2c939e7934e3, State: Running, Role: LEADER
I20260924 07:22:34.064188 25566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:34.064268 25570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8acc4e75974928898e2c939e7934e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260924 07:22:34.064738 25572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c8acc4e75974928898e2c939e7934e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8acc4e75974928898e2c939e7934e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } } }
I20260924 07:22:34.064832 25572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:34.065212 25573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c8acc4e75974928898e2c939e7934e3. Latest consensus state: current_term: 1 leader_uuid: "8c8acc4e75974928898e2c939e7934e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8acc4e75974928898e2c939e7934e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } } }
I20260924 07:22:34.065294 25573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:34.065763 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:34.063893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:60773 (local address 127.0.0.1:45911)
0924 07:22:34.064073 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:34.064082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:34.064114 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:22:34.064228 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:34.064235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:34.064268 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:34.064450 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:34.065123 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:34.065127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:34.065128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:34.065132 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:34.065177 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:34.065193 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:34.065474 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:34.065500 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 07:22:34.065533 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260924 07:22:34.065886 25575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:34.066344 25575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:34.066783 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:34.067978 25575 catalog_manager.cc:1347] Generated new cluster ID: 6a02666ff11241a3ad76acb9df75b4db
I20260924 07:22:34.068027 25575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:34.073716 25575 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:34.073860 25575 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:34.075930 25575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c8acc4e75974928898e2c939e7934e3: Generated new TSK 0
I20260924 07:22:34.076086 25575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:34.076376 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:34.070023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:50921 (local address 127.0.0.1:45911)
0924 07:22:34.070057 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:34.070058 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:34.073420 (+ 3362us) server_negotiation.cc:407] Connection header received
0924 07:22:34.073612 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:34.073613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:34.073618 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:34.073664 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:34.075935 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:34.075937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:34.075937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:34.075942 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:34.075957 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:34.075961 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:34.076001 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:34.076285 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 07:22:34.076302 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 07:22:34.077729 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 25498
2026-09-24T07:22:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:34 dist-test-slave-rc1w krb5kdc[25597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:34 dist-test-slave-rc1w krb5kdc[25597](info): set up 2 sockets
Sep 24 07:22:34 dist-test-slave-rc1w krb5kdc[25597](info): commencing operation
krb5kdc: starting...
W20260924 07:22:35.109884 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:35 dist-test-slave-rc1w krb5kdc[25597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:22:35.223802 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:41869
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:41869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:35.302271 25610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:35.302460 25610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:35.302482 25610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:35.303531 25610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:35.303565 25610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:35.303578 25610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:35.303591 25610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:35.303602 25610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:35.304932 25610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:41869
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:41869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:35.305107 25610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:35.305270 25610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:35.309837 25615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:35.309823 25616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:35.309993 25610 server_base.cc:1034] running on GCE node
W20260924 07:22:35.309819 25618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:35.310271 25610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:35.310545 25610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:35.311700 25610 hybrid_clock.cc:648] HybridClock initialized: now 1790234555311686 us; error 42 us; skew 500 ppm
Sep 24 07:22:35 dist-test-slave-rc1w krb5kdc[25597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234555, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:35.314000 25610 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:22:35.314410 25610 webserver.cc:460] Webserver started at http://127.3.246.126:44985/ using document root <none> and password file <none>
I20260924 07:22:35.314620 25610 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:35.314682 25610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:35.314827 25610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:35.315855 25610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "3a11975d67da49628228b0d2cec8091e"
format_stamp: "Formatted at 2026-09-24 07:22:35 on dist-test-slave-rc1w"
server_key: "641d57a2f1c6d38d7988527cb9f5262e"
server_key_iv: "38391a2ae06549dce6cd8b3e8a72cb29"
server_key_version: "encryptionkey@0"
I20260924 07:22:35.316201 25610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "3a11975d67da49628228b0d2cec8091e"
format_stamp: "Formatted at 2026-09-24 07:22:35 on dist-test-slave-rc1w"
server_key: "641d57a2f1c6d38d7988527cb9f5262e"
server_key_iv: "38391a2ae06549dce6cd8b3e8a72cb29"
server_key_version: "encryptionkey@0"
I20260924 07:22:35.317466 25610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:22:35.318214 25625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:35.318374 25610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:35.318464 25610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "3a11975d67da49628228b0d2cec8091e"
format_stamp: "Formatted at 2026-09-24 07:22:35 on dist-test-slave-rc1w"
server_key: "641d57a2f1c6d38d7988527cb9f5262e"
server_key_iv: "38391a2ae06549dce6cd8b3e8a72cb29"
server_key_version: "encryptionkey@0"
I20260924 07:22:35.318544 25610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:35.333272 25610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:35.335140 25610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:35.335299 25610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:35.339105 25610 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:41869
I20260924 07:22:35.339136 25677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:41869 every 8 connection(s)
I20260924 07:22:35.339479 25610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:35.340337 25678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:35.342687 25678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e: Bootstrap starting.
I20260924 07:22:35.343245 25678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:35.343516 25678 log.cc:824] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:35.344213 25678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e: No bootstrap required, opened a new log
I20260924 07:22:35.345780 25678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11975d67da49628228b0d2cec8091e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41869 } }
I20260924 07:22:35.345916 25678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:35.345965 25678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a11975d67da49628228b0d2cec8091e, State: Initialized, Role: FOLLOWER
I20260924 07:22:35.346115 25678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11975d67da49628228b0d2cec8091e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41869 } }
I20260924 07:22:35.346210 25678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:35.346232 25678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:35.346290 25678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:35.347105 25678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11975d67da49628228b0d2cec8091e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41869 } }
I20260924 07:22:35.347210 25678 leader_election.cc:302] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a11975d67da49628228b0d2cec8091e; no voters:
I20260924 07:22:35.347376 25678 leader_election.cc:288] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:35.347402 25681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:35.347554 25681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [term 1 LEADER]: Becoming Leader. State: Replica: 3a11975d67da49628228b0d2cec8091e, State: Running, Role: LEADER
I20260924 07:22:35.347620 25678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:35.347684 25681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11975d67da49628228b0d2cec8091e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41869 } }
I20260924 07:22:35.348017 25682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a11975d67da49628228b0d2cec8091e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11975d67da49628228b0d2cec8091e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41869 } } }
I20260924 07:22:35.348114 25682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:35.348028 25683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a11975d67da49628228b0d2cec8091e. Latest consensus state: current_term: 1 leader_uuid: "3a11975d67da49628228b0d2cec8091e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11975d67da49628228b0d2cec8091e" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41869 } } }
I20260924 07:22:35.348249 25683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:35.348407 25690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:35.348426 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 25610
I20260924 07:22:35.348546 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:35.348759 4057 external_mini_cluster.cc:1428] Setting key 4e377d88dbecf9a753a2785693df0c04
I20260924 07:22:35.348937 25690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:35.350880 25690 catalog_manager.cc:1347] Generated new cluster ID: 955b30d3709a448190a32f440ba6089d
I20260924 07:22:35.350932 25690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:22:35 dist-test-slave-rc1w krb5kdc[25597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:35.360915 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:35.349726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43305 (local address 127.3.246.126:41869)
0924 07:22:35.349882 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:35.349892 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:35.349925 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:22:35.350056 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:35.350063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:35.350117 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:35.350254 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:35.351061 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:35.352457 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:35.353499 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:35.354134 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:35.354260 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:35.357073 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:35.357077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:35.357078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:35.357089 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:35.359012 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:35.359470 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:35.359472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:35.359473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:35.359581 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:35.359829 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:35.359830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:35.359830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:35.360127 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:35.360267 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:35.360431 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 07:22:35.360536 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260924 07:22:35.360957 25690 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:35.361734 25690 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:35.362039 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:35.369725 25690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a11975d67da49628228b0d2cec8091e: Generated new TSK 0
I20260924 07:22:35.369908 25690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:35.370605 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:35.363399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55371 (local address 127.3.246.126:41869)
0924 07:22:35.363428 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:35.363429 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:35.363541 (+ 112us) server_negotiation.cc:407] Connection header received
0924 07:22:35.363622 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:35.363622 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:35.363630 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:35.363655 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:35.364264 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:35.365012 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:35.366003 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:35.366341 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:35.366376 (+ 35us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:35.367724 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:35.367725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:35.367726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:35.367732 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:35.369193 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:35.369753 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:35.369754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:35.369754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:35.369809 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:35.370081 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:35.370082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:35.370082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:35.370191 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:35.370247 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:35.370439 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 07:22:35.370505 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":14}
I20260924 07:22:35.372666 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 25610
W20260924 07:22:35.376713 25595 connection.cc:535] client connection to 127.3.246.126:41869 recv error: Network error: failed to read from TLS socket (remote: 127.3.246.126:41869): Connection reset by peer (error 104)
2026-09-24T07:22:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:35 dist-test-slave-rc1w krb5kdc[25709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:35 dist-test-slave-rc1w krb5kdc[25709](info): set up 2 sockets
Sep 24 07:22:35 dist-test-slave-rc1w krb5kdc[25709](info): commencing operation
krb5kdc: starting...
W20260924 07:22:36.397917 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:36 dist-test-slave-rc1w krb5kdc[25709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:22:36.517046 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:33783
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:32935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:33783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:36.597621 25722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:36.597904 25722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:36.597929 25722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:36.599083 25722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:36.599126 25722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:36.599148 25722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:36.599161 25722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:36.599174 25722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:36.600622 25722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:32935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:33783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:33783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:36.600860 25722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:36.601100 25722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:36.605953 25727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:36.605964 25728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:36.606076 25730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:36.606330 25722 server_base.cc:1034] running on GCE node
I20260924 07:22:36.606580 25722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:36.606860 25722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:36.608047 25722 hybrid_clock.cc:648] HybridClock initialized: now 1790234556608032 us; error 28 us; skew 500 ppm
Sep 24 07:22:36 dist-test-slave-rc1w krb5kdc[25709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234556, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:36.625203 25722 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:22:36.625778 25722 webserver.cc:460] Webserver started at http://127.3.246.126:35955/ using document root <none> and password file <none>
I20260924 07:22:36.626012 25722 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:36.626078 25722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:36.626217 25722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:36.627310 25722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "0a033495381041fdbe42315a4a267441"
format_stamp: "Formatted at 2026-09-24 07:22:36 on dist-test-slave-rc1w"
server_key: "df5c3ad6296c7b346e071ee4a479a129"
server_key_iv: "2f5c2a6daf405a25357b6d7a583a927a"
server_key_version: "encryptionkey@0"
I20260924 07:22:36.627686 25722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "0a033495381041fdbe42315a4a267441"
format_stamp: "Formatted at 2026-09-24 07:22:36 on dist-test-slave-rc1w"
server_key: "df5c3ad6296c7b346e071ee4a479a129"
server_key_iv: "2f5c2a6daf405a25357b6d7a583a927a"
server_key_version: "encryptionkey@0"
I20260924 07:22:36.629196 25722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:22:36.629963 25737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:36.630137 25722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:22:36.630244 25722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "0a033495381041fdbe42315a4a267441"
format_stamp: "Formatted at 2026-09-24 07:22:36 on dist-test-slave-rc1w"
server_key: "df5c3ad6296c7b346e071ee4a479a129"
server_key_iv: "2f5c2a6daf405a25357b6d7a583a927a"
server_key_version: "encryptionkey@0"
I20260924 07:22:36.630316 25722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:36.652771 25722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:36.655293 25722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:36.655464 25722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:36.659492 25722 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:33783
I20260924 07:22:36.659534 25789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:33783 every 8 connection(s)
I20260924 07:22:36.659922 25722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:36.660852 25790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:36.661863 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 25722
I20260924 07:22:36.661994 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:36.662192 4057 external_mini_cluster.cc:1428] Setting key f57610fc0346511e442d34ce8e538b03
I20260924 07:22:36.663713 25790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441: Bootstrap starting.
I20260924 07:22:36.664387 25790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:36.664669 25790 log.cc:824] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:36.665431 25790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441: No bootstrap required, opened a new log
I20260924 07:22:36.667754 25790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a033495381041fdbe42315a4a267441" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33783 } }
I20260924 07:22:36.667936 25790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:36.668009 25790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a033495381041fdbe42315a4a267441, State: Initialized, Role: FOLLOWER
I20260924 07:22:36.668207 25793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:36.662756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52231 (local address 127.3.246.126:33783)
0924 07:22:36.662930 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:36.662939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:36.662974 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:22:36.663564 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:36.663571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:36.663636 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:36.663774 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:36.664019 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:36.665161 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:36.666246 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:36.666649 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:36.666727 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:36.667486 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:36.667489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:36.667490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:36.667649 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:36.667677 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:36.667688 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:36.667751 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:36.667903 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 07:22:36.667937 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260924 07:22:36.668183 25790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a033495381041fdbe42315a4a267441" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33783 } }
I20260924 07:22:36.668274 25790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:36.668313 25790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:36.668359 25790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:36.669044 25790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a033495381041fdbe42315a4a267441" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33783 } }
I20260924 07:22:36.669162 25790 leader_election.cc:302] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a033495381041fdbe42315a4a267441; no voters:
I20260924 07:22:36.669211 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:36.669332 25790 leader_election.cc:288] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:36.669359 25795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:36.669514 25795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [term 1 LEADER]: Becoming Leader. State: Replica: 0a033495381041fdbe42315a4a267441, State: Running, Role: LEADER
I20260924 07:22:36.669564 25790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:36.669654 25795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a033495381041fdbe42315a4a267441" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33783 } }
I20260924 07:22:36.670100 25796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a033495381041fdbe42315a4a267441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a033495381041fdbe42315a4a267441" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33783 } } }
I20260924 07:22:36.670173 25796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:36.670392 25797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a033495381041fdbe42315a4a267441. Latest consensus state: current_term: 1 leader_uuid: "0a033495381041fdbe42315a4a267441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a033495381041fdbe42315a4a267441" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33783 } } }
I20260924 07:22:36.670737 25797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441 [sys.catalog]: This master's current role is: LEADER
W20260924 07:22:36.671350 25815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:22:36.671449 25815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:22:36.671555 25805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:36.671749 25805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:36.673126 25805 catalog_manager.cc:1347] Generated new cluster ID: 3641ae91b10b486f946a63c2c440a6ad
I20260924 07:22:36.673177 25805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:36.674474 25793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:36.670808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59865 (local address 127.3.246.126:33783)
0924 07:22:36.670843 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:36.670845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:36.671175 (+ 330us) server_negotiation.cc:407] Connection header received
0924 07:22:36.671245 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:36.671245 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:36.671253 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:36.671321 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:36.671495 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:36.672186 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:36.673199 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:36.673481 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:36.673547 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:36.674227 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:36.674228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:36.674229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:36.674233 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:36.674246 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:36.674249 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:36.674306 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:36.674400 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 07:22:36.674408 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:22:36.685614 25805 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:36.686395 25805 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:36.690310 25805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a033495381041fdbe42315a4a267441: Generated new TSK 0
I20260924 07:22:36.690493 25805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:36.695787 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 25722
W20260924 07:22:36.699625 25707 connection.cc:535] client connection to 127.3.246.126:33783 recv error: Network error: failed to read from TLS socket (remote: 127.3.246.126:33783): Connection reset by peer (error 104)
2026-09-24T07:22:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:36 dist-test-slave-rc1w krb5kdc[25821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:36 dist-test-slave-rc1w krb5kdc[25821](info): set up 2 sockets
Sep 24 07:22:36 dist-test-slave-rc1w krb5kdc[25821](info): commencing operation
krb5kdc: starting...
W20260924 07:22:37.729871 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:37 dist-test-slave-rc1w krb5kdc[25821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:22:37.845330 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37899
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:37899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:37.924793 25834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:37.925024 25834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:37.925062 25834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:37.926292 25834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:37.926357 25834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:37.926383 25834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:37.926414 25834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:37.926443 25834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:37.927906 25834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:37899
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:37899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:37.928179 25834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:37.928586 25834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:22:37.933503 25834 server_base.cc:1034] running on GCE node
W20260924 07:22:37.933413 25839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:37.933413 25842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:37.933439 25840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:37.933890 25834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:37.934209 25834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:37.935400 25834 hybrid_clock.cc:648] HybridClock initialized: now 1790234557935390 us; error 39 us; skew 500 ppm
Sep 24 07:22:37 dist-test-slave-rc1w krb5kdc[25821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234557, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:37.937968 25834 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:22:37.938403 25834 webserver.cc:460] Webserver started at http://127.3.246.126:33165/ using document root <none> and password file <none>
I20260924 07:22:37.938625 25834 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:37.938678 25834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:37.938766 25834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:37.939759 25834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "26b714a139ae44788b67ec28c21dbe1d"
format_stamp: "Formatted at 2026-09-24 07:22:37 on dist-test-slave-rc1w"
server_key: "2c733efe1785b57be7370dfa55290dc1"
server_key_iv: "db780928d7b0c05b46f32641a444a0be"
server_key_version: "encryptionkey@0"
I20260924 07:22:37.940053 25834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "26b714a139ae44788b67ec28c21dbe1d"
format_stamp: "Formatted at 2026-09-24 07:22:37 on dist-test-slave-rc1w"
server_key: "2c733efe1785b57be7370dfa55290dc1"
server_key_iv: "db780928d7b0c05b46f32641a444a0be"
server_key_version: "encryptionkey@0"
I20260924 07:22:37.941301 25834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 07:22:37.942073 25849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:37.942269 25834 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:37.942355 25834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "26b714a139ae44788b67ec28c21dbe1d"
format_stamp: "Formatted at 2026-09-24 07:22:37 on dist-test-slave-rc1w"
server_key: "2c733efe1785b57be7370dfa55290dc1"
server_key_iv: "db780928d7b0c05b46f32641a444a0be"
server_key_version: "encryptionkey@0"
I20260924 07:22:37.942410 25834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:37.945487 25834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:37.947346 25834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:37.947448 25834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:37.951057 25834 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:37899
I20260924 07:22:37.951118 25901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:37899 every 8 connection(s)
I20260924 07:22:37.951408 25834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:37.952382 25902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:37.954656 25902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d: Bootstrap starting.
I20260924 07:22:37.955219 25902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:37.955492 25902 log.cc:824] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:37.956185 25902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d: No bootstrap required, opened a new log
I20260924 07:22:37.957775 25902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b714a139ae44788b67ec28c21dbe1d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37899 } }
I20260924 07:22:37.957922 25902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:37.957966 25902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b714a139ae44788b67ec28c21dbe1d, State: Initialized, Role: FOLLOWER
I20260924 07:22:37.958127 25902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b714a139ae44788b67ec28c21dbe1d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37899 } }
I20260924 07:22:37.958205 25902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:37.958247 25902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:37.958300 25902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:37.959089 25902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b714a139ae44788b67ec28c21dbe1d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37899 } }
I20260924 07:22:37.959205 25902 leader_election.cc:302] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26b714a139ae44788b67ec28c21dbe1d; no voters:
I20260924 07:22:37.959414 25902 leader_election.cc:288] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:37.959438 25905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:37.959511 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 25834
I20260924 07:22:37.959563 25905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [term 1 LEADER]: Becoming Leader. State: Replica: 26b714a139ae44788b67ec28c21dbe1d, State: Running, Role: LEADER
I20260924 07:22:37.959625 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:37.959673 25902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:37.959681 25905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b714a139ae44788b67ec28c21dbe1d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37899 } }
I20260924 07:22:37.959825 4057 external_mini_cluster.cc:1428] Setting key 065914d43daf9f51cd1d27d07f0327eb
I20260924 07:22:37.960019 25905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26b714a139ae44788b67ec28c21dbe1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b714a139ae44788b67ec28c21dbe1d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37899 } } }
I20260924 07:22:37.960096 25905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:37.960078 25906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26b714a139ae44788b67ec28c21dbe1d. Latest consensus state: current_term: 1 leader_uuid: "26b714a139ae44788b67ec28c21dbe1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b714a139ae44788b67ec28c21dbe1d" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 37899 } } }
I20260924 07:22:37.960177 25906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:37.960572 25911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:37.961088 25911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:37.962846 25911 catalog_manager.cc:1347] Generated new cluster ID: 61c8a59e9b7643c9b651cbf478147fc0
I20260924 07:22:37.962903 25911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:37.962989 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:37.960438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58503 (local address 127.3.246.126:37899)
0924 07:22:37.960985 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:37.960995 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:37.961264 (+ 269us) server_negotiation.cc:407] Connection header received
0924 07:22:37.961389 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:37.961398 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:37.961431 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:37.961567 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:37.962481 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:37.962484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:37.962485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:37.962489 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:37.962532 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:37.962546 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:37.962618 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:37.962760 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 07:22:37.962800 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":61,"threads_started":1}
I20260924 07:22:37.963876 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:37.969686 25911 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:37.969838 25911 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:37.970618 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:37.965367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53673 (local address 127.3.246.126:37899)
0924 07:22:37.965401 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:37.965404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:37.969508 (+ 4104us) server_negotiation.cc:407] Connection header received
0924 07:22:37.969595 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:37.969596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:37.969600 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:37.969685 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:37.970307 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:37.970308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:37.970309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:37.970313 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:37.970328 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:37.970333 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:37.970388 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:37.970459 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 07:22:37.970472 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 07:22:37.980041 25911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26b714a139ae44788b67ec28c21dbe1d: Generated new TSK 0
I20260924 07:22:37.980247 25911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:37.983806 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 25834
2026-09-24T07:22:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:38 dist-test-slave-rc1w krb5kdc[25933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:38 dist-test-slave-rc1w krb5kdc[25933](info): set up 2 sockets
Sep 24 07:22:38 dist-test-slave-rc1w krb5kdc[25933](info): commencing operation
krb5kdc: starting...
W20260924 07:22:39.021905 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:39 dist-test-slave-rc1w krb5kdc[25933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:22:39.140007 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:41273
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:41273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:39.218605 25946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:39.218812 25946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:39.218834 25946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:39.219964 25946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:39.220005 25946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:39.220019 25946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:39.220031 25946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:39.220043 25946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:39.221549 25946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:41273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:41273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:39.221823 25946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:39.222048 25946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:39.226815 25954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:39.226827 25951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:39.226882 25952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:39.226946 25946 server_base.cc:1034] running on GCE node
I20260924 07:22:39.227321 25946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:39.227595 25946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:39.228744 25946 hybrid_clock.cc:648] HybridClock initialized: now 1790234559228734 us; error 30 us; skew 500 ppm
Sep 24 07:22:39 dist-test-slave-rc1w krb5kdc[25933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234559, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:39.231142 25946 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:22:39.231554 25946 webserver.cc:460] Webserver started at http://127.3.246.126:42561/ using document root <none> and password file <none>
I20260924 07:22:39.231776 25946 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:39.231820 25946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:39.231981 25946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:39.233024 25946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "ef3356d531f14162b95732dd4a1043d7"
format_stamp: "Formatted at 2026-09-24 07:22:39 on dist-test-slave-rc1w"
server_key: "621acb40a488a3ff75e75b158407bba9"
server_key_iv: "c5d8836e7f8607de94a3642c062aa105"
server_key_version: "encryptionkey@0"
I20260924 07:22:39.233381 25946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "ef3356d531f14162b95732dd4a1043d7"
format_stamp: "Formatted at 2026-09-24 07:22:39 on dist-test-slave-rc1w"
server_key: "621acb40a488a3ff75e75b158407bba9"
server_key_iv: "c5d8836e7f8607de94a3642c062aa105"
server_key_version: "encryptionkey@0"
I20260924 07:22:39.234948 25946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:22:39.235733 25961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:39.235939 25946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:22:39.236027 25946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "ef3356d531f14162b95732dd4a1043d7"
format_stamp: "Formatted at 2026-09-24 07:22:39 on dist-test-slave-rc1w"
server_key: "621acb40a488a3ff75e75b158407bba9"
server_key_iv: "c5d8836e7f8607de94a3642c062aa105"
server_key_version: "encryptionkey@0"
I20260924 07:22:39.236104 25946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:39.246315 25946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:39.248003 25946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:39.248162 25946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:39.252000 25946 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:41273
I20260924 07:22:39.252045 26013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:41273 every 8 connection(s)
I20260924 07:22:39.252401 25946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:39.253222 26014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:39.254709 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 25946
I20260924 07:22:39.254798 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:39.255005 4057 external_mini_cluster.cc:1428] Setting key 4830e16a8ea289d55fcd713fae2d9183
I20260924 07:22:39.255797 26014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7: Bootstrap starting.
I20260924 07:22:39.256372 26014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:39.256659 26014 log.cc:824] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:39.257320 26014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7: No bootstrap required, opened a new log
I20260924 07:22:39.259100 26014 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3356d531f14162b95732dd4a1043d7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41273 } }
I20260924 07:22:39.259264 26014 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:39.259315 26014 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3356d531f14162b95732dd4a1043d7, State: Initialized, Role: FOLLOWER
I20260924 07:22:39.259449 26014 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3356d531f14162b95732dd4a1043d7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41273 } }
I20260924 07:22:39.259543 26014 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:39.259586 26014 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:39.259634 26014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:39.260294 26014 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3356d531f14162b95732dd4a1043d7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41273 } }
I20260924 07:22:39.260413 26014 leader_election.cc:302] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef3356d531f14162b95732dd4a1043d7; no voters:
I20260924 07:22:39.260566 26014 leader_election.cc:288] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:39.260587 26019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:39.260710 26019 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [term 1 LEADER]: Becoming Leader. State: Replica: ef3356d531f14162b95732dd4a1043d7, State: Running, Role: LEADER
I20260924 07:22:39.260831 26014 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:39.260845 26019 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3356d531f14162b95732dd4a1043d7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41273 } }
I20260924 07:22:39.261286 26021 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef3356d531f14162b95732dd4a1043d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3356d531f14162b95732dd4a1043d7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41273 } } }
I20260924 07:22:39.261507 26021 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:39.261447 26020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef3356d531f14162b95732dd4a1043d7. Latest consensus state: current_term: 1 leader_uuid: "ef3356d531f14162b95732dd4a1043d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3356d531f14162b95732dd4a1043d7" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 41273 } } }
I20260924 07:22:39.261768 26020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7 [sys.catalog]: This master's current role is: LEADER
Sep 24 07:22:39 dist-test-slave-rc1w krb5kdc[25933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:39.261997 26028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:39.262497 26028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:39.264487 26028 catalog_manager.cc:1347] Generated new cluster ID: d6240d1ea9654b0b9bc807036d7d6d80
I20260924 07:22:39.264582 26028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:39.266112 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:39.255520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53089 (local address 127.3.246.126:41273)
0924 07:22:39.255719 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:39.255730 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:39.255771 (+ 41us) server_negotiation.cc:407] Connection header received
0924 07:22:39.256228 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:39.256234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:39.256280 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:39.256428 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:39.257103 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:39.258172 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:39.258914 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:39.259375 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:39.259487 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:39.262450 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:39.262454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:39.262454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:39.262466 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:39.264474 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:39.264886 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:39.264889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:39.264889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:39.264985 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:39.265205 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:39.265205 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:39.265206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:39.265469 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:39.265600 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:39.265801 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 07:22:39.265922 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260924 07:22:39.267083 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:39.275324 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:39.268609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54203 (local address 127.3.246.126:41273)
0924 07:22:39.268653 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:39.268656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:39.268665 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:22:39.268687 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:39.268688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:39.268699 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:39.268762 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:39.269338 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:39.270082 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:39.271029 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:39.271330 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:39.271403 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:39.272640 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:39.272641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:39.272642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:39.272648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:39.274200 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:39.274564 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:39.274566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:39.274566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:39.274623 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:39.274820 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:39.274821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:39.274821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:39.274959 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:39.275053 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:39.275159 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 07:22:39.275234 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:22:39.291487 26028 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:39.292028 26028 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:39.299883 26028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef3356d531f14162b95732dd4a1043d7: Generated new TSK 0
I20260924 07:22:39.300086 26028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:39.305567 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 25946
2026-09-24T07:22:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:39 dist-test-slave-rc1w krb5kdc[26045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:39 dist-test-slave-rc1w krb5kdc[26045](info): set up 2 sockets
Sep 24 07:22:39 dist-test-slave-rc1w krb5kdc[26045](info): commencing operation
krb5kdc: starting...
W20260924 07:22:40.341827 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:40 dist-test-slave-rc1w krb5kdc[26045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:22:40.460295 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:38317
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:38317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:40.543247 26058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:40.543488 26058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:40.543526 26058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:40.544701 26058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:40.544770 26058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:40.544795 26058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:40.544827 26058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:40.544850 26058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:40.546399 26058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:38317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:38317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:40.546672 26058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:40.546919 26058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:40.551800 26066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:40.551877 26063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:40.551951 26058 server_base.cc:1034] running on GCE node
W20260924 07:22:40.551882 26064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:40.552269 26058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:40.552558 26058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:40.553741 26058 hybrid_clock.cc:648] HybridClock initialized: now 1790234560553735 us; error 35 us; skew 500 ppm
Sep 24 07:22:40 dist-test-slave-rc1w krb5kdc[26045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234560, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:40.556250 26058 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:22:40.556641 26058 webserver.cc:460] Webserver started at http://127.3.246.126:38907/ using document root <none> and password file <none>
I20260924 07:22:40.556841 26058 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:40.556882 26058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:40.556967 26058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:40.558068 26058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "22c75c2272e54a5a9aec1c1832edb511"
format_stamp: "Formatted at 2026-09-24 07:22:40 on dist-test-slave-rc1w"
server_key: "ac50e9080943c9f4359500a11c2f0732"
server_key_iv: "5a878473c4d4ca82e26622806fb4ab3c"
server_key_version: "encryptionkey@0"
I20260924 07:22:40.558383 26058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "22c75c2272e54a5a9aec1c1832edb511"
format_stamp: "Formatted at 2026-09-24 07:22:40 on dist-test-slave-rc1w"
server_key: "ac50e9080943c9f4359500a11c2f0732"
server_key_iv: "5a878473c4d4ca82e26622806fb4ab3c"
server_key_version: "encryptionkey@0"
I20260924 07:22:40.559618 26058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:22:40.560312 26073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:40.560505 26058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:40.560572 26058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "22c75c2272e54a5a9aec1c1832edb511"
format_stamp: "Formatted at 2026-09-24 07:22:40 on dist-test-slave-rc1w"
server_key: "ac50e9080943c9f4359500a11c2f0732"
server_key_iv: "5a878473c4d4ca82e26622806fb4ab3c"
server_key_version: "encryptionkey@0"
I20260924 07:22:40.560664 26058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:40.572103 26058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:40.574178 26058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:40.574337 26058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:40.578238 26058 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:38317
I20260924 07:22:40.578289 26125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:38317 every 8 connection(s)
I20260924 07:22:40.578648 26058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:40.579476 26126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:40.581768 26126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511: Bootstrap starting.
I20260924 07:22:40.582576 26126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:40.582962 26126 log.cc:824] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:40.583662 26126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511: No bootstrap required, opened a new log
I20260924 07:22:40.585093 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 26058
I20260924 07:22:40.585189 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:40.585392 4057 external_mini_cluster.cc:1428] Setting key 867ac3222369e3de1fbf2a8b36052d18
I20260924 07:22:40.585325 26126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c75c2272e54a5a9aec1c1832edb511" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 38317 } }
I20260924 07:22:40.585500 26126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:40.585553 26126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c75c2272e54a5a9aec1c1832edb511, State: Initialized, Role: FOLLOWER
I20260924 07:22:40.585733 26126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c75c2272e54a5a9aec1c1832edb511" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 38317 } }
I20260924 07:22:40.585830 26126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:40.585901 26126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:40.585960 26126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:40.586972 26126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c75c2272e54a5a9aec1c1832edb511" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 38317 } }
I20260924 07:22:40.587093 26126 leader_election.cc:302] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22c75c2272e54a5a9aec1c1832edb511; no voters:
I20260924 07:22:40.587271 26126 leader_election.cc:288] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:40.587323 26131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:40.587467 26131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [term 1 LEADER]: Becoming Leader. State: Replica: 22c75c2272e54a5a9aec1c1832edb511, State: Running, Role: LEADER
I20260924 07:22:40.587538 26126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:40.587597 26131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c75c2272e54a5a9aec1c1832edb511" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 38317 } }
I20260924 07:22:40.588003 26131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22c75c2272e54a5a9aec1c1832edb511. Latest consensus state: current_term: 1 leader_uuid: "22c75c2272e54a5a9aec1c1832edb511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c75c2272e54a5a9aec1c1832edb511" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 38317 } } }
I20260924 07:22:40.588121 26131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:40.588141 26132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22c75c2272e54a5a9aec1c1832edb511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c75c2272e54a5a9aec1c1832edb511" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 38317 } } }
I20260924 07:22:40.588233 26132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:40.588474 26140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:40.589175 26140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:40.590965 26140 catalog_manager.cc:1347] Generated new cluster ID: d19f9d788337487eb798e28893c99709
I20260924 07:22:40.591017 26140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:40.591636 26130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:40.585879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45623 (local address 127.3.246.126:38317)
0924 07:22:40.586234 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:40.586242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:40.586265 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:22:40.586363 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:40.586369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:40.586410 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:40.586518 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:40.586789 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:40.587976 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:40.589168 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:40.589865 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:40.589975 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:40.590731 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:40.590734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:40.590735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:40.590936 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:40.590973 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:40.590989 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:40.591316 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:40.591350 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 07:22:40.591416 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":93,"threads_started":1}
I20260924 07:22:40.592691 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:40.599005 26140 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:40.599750 26140 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:40.602358 26140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22c75c2272e54a5a9aec1c1832edb511: Generated new TSK 0
I20260924 07:22:40.602576 26140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:40.602802 26130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:40.599098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35153 (local address 127.3.246.126:38317)
0924 07:22:40.599138 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:40.599140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:40.599358 (+ 218us) server_negotiation.cc:407] Connection header received
0924 07:22:40.599390 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:40.599391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:40.599403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:40.599480 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:40.599620 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:40.600384 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:40.601380 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:40.601720 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:40.601794 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:40.602521 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:40.602522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:40.602523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:40.602528 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:40.602544 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:40.602549 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:40.602610 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:40.602683 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 07:22:40.602722 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:22:40.604908 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 26058
2026-09-24T07:22:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:40 dist-test-slave-rc1w krb5kdc[26157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:40 dist-test-slave-rc1w krb5kdc[26157](info): set up 2 sockets
Sep 24 07:22:40 dist-test-slave-rc1w krb5kdc[26157](info): commencing operation
krb5kdc: starting...
W20260924 07:22:41.645881 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:41 dist-test-slave-rc1w krb5kdc[26157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:22:41.761153 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:33369
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:33369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:41.839949 26170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:41.840324 26170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:41.840406 26170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:41.841534 26170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:41.841571 26170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:41.841585 26170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:41.841598 26170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:41.841609 26170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:41.842988 26170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:33369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:33369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:41.843173 26170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:41.843343 26170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:41.848279 26175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:41.848312 26178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:41.848475 26176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:41.848672 26170 server_base.cc:1034] running on GCE node
I20260924 07:22:41.848924 26170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:41.849249 26170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:41.850406 26170 hybrid_clock.cc:648] HybridClock initialized: now 1790234561850397 us; error 44 us; skew 500 ppm
Sep 24 07:22:41 dist-test-slave-rc1w krb5kdc[26157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234561, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:41.862064 26170 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:22:41.862583 26170 webserver.cc:460] Webserver started at http://127.3.246.126:34825/ using document root <none> and password file <none>
I20260924 07:22:41.862825 26170 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:41.862892 26170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:41.863044 26170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:41.864109 26170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "032a242c91d145699f905d1e22bbb0c2"
format_stamp: "Formatted at 2026-09-24 07:22:41 on dist-test-slave-rc1w"
server_key: "75abf151a03bd2c2a20612341adc74e1"
server_key_iv: "b5f3990bd2bdde19d9a290d8e19a0843"
server_key_version: "encryptionkey@0"
I20260924 07:22:41.864467 26170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "032a242c91d145699f905d1e22bbb0c2"
format_stamp: "Formatted at 2026-09-24 07:22:41 on dist-test-slave-rc1w"
server_key: "75abf151a03bd2c2a20612341adc74e1"
server_key_iv: "b5f3990bd2bdde19d9a290d8e19a0843"
server_key_version: "encryptionkey@0"
I20260924 07:22:41.865801 26170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:22:41.866528 26185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:41.866736 26170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:22:41.866843 26170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "032a242c91d145699f905d1e22bbb0c2"
format_stamp: "Formatted at 2026-09-24 07:22:41 on dist-test-slave-rc1w"
server_key: "75abf151a03bd2c2a20612341adc74e1"
server_key_iv: "b5f3990bd2bdde19d9a290d8e19a0843"
server_key_version: "encryptionkey@0"
I20260924 07:22:41.866931 26170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:41.870224 26170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:41.871987 26170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:41.872138 26170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:41.875998 26170 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:33369
I20260924 07:22:41.876026 26237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:33369 every 8 connection(s)
I20260924 07:22:41.876340 26170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:41.877247 26238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:41.879432 26238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2: Bootstrap starting.
I20260924 07:22:41.879949 26238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:41.880170 26238 log.cc:824] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:41.880779 26238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2: No bootstrap required, opened a new log
I20260924 07:22:41.882366 26238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a242c91d145699f905d1e22bbb0c2" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33369 } }
I20260924 07:22:41.882496 26238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:22:41.882519 26238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a242c91d145699f905d1e22bbb0c2, State: Initialized, Role: FOLLOWER
I20260924 07:22:41.882648 26238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a242c91d145699f905d1e22bbb0c2" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33369 } }
I20260924 07:22:41.882723 26238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:22:41.882751 26238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:22:41.882779 26238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:41.883525 26238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a242c91d145699f905d1e22bbb0c2" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33369 } }
I20260924 07:22:41.883610 26238 leader_election.cc:302] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 032a242c91d145699f905d1e22bbb0c2; no voters:
I20260924 07:22:41.883723 26238 leader_election.cc:288] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:22:41.883800 26241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:41.883955 26241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 032a242c91d145699f905d1e22bbb0c2, State: Running, Role: LEADER
I20260924 07:22:41.884027 26238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:41.884080 26241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a242c91d145699f905d1e22bbb0c2" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33369 } }
I20260924 07:22:41.884526 26242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "032a242c91d145699f905d1e22bbb0c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a242c91d145699f905d1e22bbb0c2" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33369 } } }
I20260924 07:22:41.884562 26243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032a242c91d145699f905d1e22bbb0c2. Latest consensus state: current_term: 1 leader_uuid: "032a242c91d145699f905d1e22bbb0c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a242c91d145699f905d1e22bbb0c2" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 33369 } } }
I20260924 07:22:41.884701 26243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:41.884882 26242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2 [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:41.885064 26250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:41.885620 26250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:41.885715 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 26170
I20260924 07:22:41.885842 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:41.886022 4057 external_mini_cluster.cc:1428] Setting key 5f81db7b8a11f8e8882c381e30f65ecb
I20260924 07:22:41.887811 26250 catalog_manager.cc:1347] Generated new cluster ID: d800c906f44e47169e1166383a6a5a8d
I20260924 07:22:41.887864 26250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:41.888326 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:41.886618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54753 (local address 127.3.246.126:33369)
0924 07:22:41.886889 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:41.886898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:41.886931 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:22:41.887042 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:41.887056 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:41.887078 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:41.887240 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:41.887885 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:41.887888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:41.887889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:41.887893 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:41.887941 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:41.887955 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:41.888020 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:41.888115 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 07:22:41.888163 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20260924 07:22:41.889163 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:41.894668 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:41.893744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54651 (local address 127.3.246.126:33369)
0924 07:22:41.893781 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:41.893782 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:41.893791 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:22:41.893812 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:41.893813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:41.893817 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:41.893881 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:41.894450 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:41.894451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:41.894451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:22:41.894455 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:41.894469 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:22:41.894473 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:41.894533 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:41.894595 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 07:22:41.894607 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:22:41.922996 26250 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:41.923193 26250 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:41.928954 26250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 032a242c91d145699f905d1e22bbb0c2: Generated new TSK 0
I20260924 07:22:41.929150 26250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:41.934643 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 26170
2026-09-24T07:22:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1330 ms)
[----------] 18 tests from AuthTokenIssuingTest (23538 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:41 dist-test-slave-rc1w krb5kdc[26269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:41 dist-test-slave-rc1w krb5kdc[26269](info): set up 2 sockets
Sep 24 07:22:41 dist-test-slave-rc1w krb5kdc[26269](info): commencing operation
krb5kdc: starting...
W20260924 07:22:42.965929 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:22:43.080559 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:46075
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:46075,127.3.246.125:39679,127.3.246.124:44869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:43.159579 26282 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:22:43.159816 26282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:43.159847 26282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:43.159909 26282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:43.161041 26282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:43.161108 26282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:43.161132 26282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:43.161160 26282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:43.161186 26282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:43.162701 26282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:46075,127.3.246.125:39679,127.3.246.124:44869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:46075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:43.162972 26282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:43.163209 26282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:43.168157 26290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:43.168156 26288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:43.168236 26282 server_base.cc:1034] running on GCE node
W20260924 07:22:43.168187 26287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:43.168614 26282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:43.168893 26282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:43.170053 26282 hybrid_clock.cc:648] HybridClock initialized: now 1790234563170040 us; error 29 us; skew 500 ppm
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:43.172456 26282 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:22:43.172900 26282 webserver.cc:460] Webserver started at http://127.3.246.126:42745/ using document root <none> and password file <none>
I20260924 07:22:43.173143 26282 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:43.173214 26282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:43.173358 26282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:43.174446 26282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "0beb1b85d44248f89e7eaf8ecdad420c"
format_stamp: "Formatted at 2026-09-24 07:22:43 on dist-test-slave-rc1w"
server_key: "968693b9b0dfaf9604a7164e4b8c0b25"
server_key_iv: "f7cae7b8c157f3a1be20832682713afd"
server_key_version: "encryptionkey@0"
I20260924 07:22:43.174901 26282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "0beb1b85d44248f89e7eaf8ecdad420c"
format_stamp: "Formatted at 2026-09-24 07:22:43 on dist-test-slave-rc1w"
server_key: "968693b9b0dfaf9604a7164e4b8c0b25"
server_key_iv: "f7cae7b8c157f3a1be20832682713afd"
server_key_version: "encryptionkey@0"
I20260924 07:22:43.176218 26282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:22:43.176928 26297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:43.177098 26282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:43.177198 26282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "0beb1b85d44248f89e7eaf8ecdad420c"
format_stamp: "Formatted at 2026-09-24 07:22:43 on dist-test-slave-rc1w"
server_key: "968693b9b0dfaf9604a7164e4b8c0b25"
server_key_iv: "f7cae7b8c157f3a1be20832682713afd"
server_key_version: "encryptionkey@0"
I20260924 07:22:43.177275 26282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:43.192842 26282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:43.194792 26282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:43.194952 26282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:43.198745 26282 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:46075
I20260924 07:22:43.198797 26349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:46075 every 8 connection(s)
I20260924 07:22:43.199146 26282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:43.200106 26350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:43.201572 26350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:22:43.205049 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 26282
I20260924 07:22:43.205133 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:43.205370 4057 external_mini_cluster.cc:1428] Setting key bcacb9939af585bc2e8d3c6461a6210f
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:43.211463 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.202290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46075 (local address 127.0.0.1:47534)
0924 07:22:43.202447 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.202484 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:43.202924 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:43.203181 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:43.203183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:43.203432 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:43.203571 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.203574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.204532 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.204532 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.205253 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.205255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.205967 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.205968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.206025 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.206552 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:43.206560 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:43.208107 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:43.210008 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.210014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.210016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.210207 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.210460 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.210461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.210461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.210543 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.210889 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:43.210893 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:43.210951 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 07:22:43.211060 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 07:22:43.211297 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":98,"thread_start_us":89,"threads_started":1}
I20260924 07:22:43.211467 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.202317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.3.246.126:46075)
0924 07:22:43.202481 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.202488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.202560 (+ 72us) server_negotiation.cc:407] Connection header received
0924 07:22:43.203059 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.203061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.203077 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.203156 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.203661 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.204453 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.205332 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.205857 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.205973 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.208192 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.208200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.208200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.208207 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.209892 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.210304 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.210306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.210307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.210384 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.210616 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.210616 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.210617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.210818 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.210908 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.211093 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.211231 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":98,"threads_started":1}
I20260924 07:22:43.212129 26350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:22:43.212498 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.212354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:39679 (local address 127.0.0.1:60050)
0924 07:22:43.212410 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.212462 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:22:43.212652 26299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.246.125:39679: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.246.125@KRBTEST.COM; defaulting to no policy
I20260924 07:22:43.213107 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.212983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:39679 (local address 127.0.0.1:60066)
0924 07:22:43.213038 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.213070 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 07:22:43.213246 26350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.125:39679: Network error: Client connection negotiation failed: client connection to 127.3.246.125:39679: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.125.keytab.
Entry for principal kudu/127.3.246.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.125.keytab.
I20260924 07:22:43.228801 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.125:39679
--webserver_interface=127.3.246.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.125.keytab
--principal=kudu/127.3.246.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:46075,127.3.246.125:39679,127.3.246.124:44869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:22:43.262511 26350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } attempt: 1
I20260924 07:22:43.263057 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.262917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:39679 (local address 127.0.0.1:60076)
0924 07:22:43.262983 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.263027 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:22:43.263464 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.263370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:39679 (local address 127.0.0.1:60078)
0924 07:22:43.263418 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.263445 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:22:43.263577 26350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.125:39679: Network error: Client connection negotiation failed: client connection to 127.3.246.125:39679: connect: Connection refused (error 111)
W20260924 07:22:43.308264 26357 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:22:43.308454 26357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:43.308472 26357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:43.308491 26357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:43.309631 26357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:43.309698 26357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:43.309715 26357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:43.309727 26357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:43.309779 26357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:43.311249 26357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:46075,127.3.246.125:39679,127.3.246.124:44869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.125.keytab
--principal=kudu/127.3.246.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.125:39679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.246.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:43.311481 26357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:43.311693 26357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:43.316440 26362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:43.316433 26363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:43.316669 26357 server_base.cc:1034] running on GCE node
W20260924 07:22:43.316428 26365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:43.316948 26357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:43.317255 26357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:43.318411 26357 hybrid_clock.cc:648] HybridClock initialized: now 1790234563318400 us; error 43 us; skew 500 ppm
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:43.320719 26357 init.cc:377] Logged in from keytab as kudu/127.3.246.125@KRBTEST.COM (short username kudu)
I20260924 07:22:43.321163 26357 webserver.cc:460] Webserver started at http://127.3.246.125:42151/ using document root <none> and password file <none>
I20260924 07:22:43.321391 26357 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:43.321456 26357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:43.321604 26357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:43.322614 26357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/data/instance:
uuid: "137179ffac554858934fae019c6d01d8"
format_stamp: "Formatted at 2026-09-24 07:22:43 on dist-test-slave-rc1w"
server_key: "3f90583cf29e5c86f06af0ffb6b13a87"
server_key_iv: "07fb38ebd86d00b70b77d2412ae02e8c"
server_key_version: "encryptionkey@0"
I20260924 07:22:43.322995 26357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/wal/instance:
uuid: "137179ffac554858934fae019c6d01d8"
format_stamp: "Formatted at 2026-09-24 07:22:43 on dist-test-slave-rc1w"
server_key: "3f90583cf29e5c86f06af0ffb6b13a87"
server_key_iv: "07fb38ebd86d00b70b77d2412ae02e8c"
server_key_version: "encryptionkey@0"
I20260924 07:22:43.324281 26357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:22:43.324956 26372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:43.325137 26357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:43.325229 26357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/wal
uuid: "137179ffac554858934fae019c6d01d8"
format_stamp: "Formatted at 2026-09-24 07:22:43 on dist-test-slave-rc1w"
server_key: "3f90583cf29e5c86f06af0ffb6b13a87"
server_key_iv: "07fb38ebd86d00b70b77d2412ae02e8c"
server_key_version: "encryptionkey@0"
I20260924 07:22:43.325309 26357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:43.331715 26350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } attempt: 2
I20260924 07:22:43.332100 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.331965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:39679 (local address 127.0.0.1:60082)
0924 07:22:43.332021 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.332072 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 07:22:43.332422 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.332332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:39679 (local address 127.0.0.1:60084)
0924 07:22:43.332373 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.332395 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 07:22:43.332500 26350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.125:39679: Network error: Client connection negotiation failed: client connection to 127.3.246.125:39679: connect: Connection refused (error 111)
I20260924 07:22:43.343021 26357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:43.345005 26357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:43.345161 26357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:43.349141 26357 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.125:39679
I20260924 07:22:43.349172 26424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.125:39679 every 8 connection(s)
I20260924 07:22:43.349510 26357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/data/info.pb
I20260924 07:22:43.350483 26425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:43.352048 26425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:22:43.353219 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 26357
I20260924 07:22:43.353354 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-1/wal/instance
I20260924 07:22:43.353554 4057 external_mini_cluster.cc:1428] Setting key 15ba7216d8b476acda40dad59c9b10ad
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.125@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.246.124@KRBTEST.COM; defaulting to no policy
I20260924 07:22:43.361477 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.352739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.3.246.126:46075)
0924 07:22:43.352799 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.352805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.352928 (+ 123us) server_negotiation.cc:407] Connection header received
0924 07:22:43.353567 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.353568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.353582 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.353701 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.354323 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.354804 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.355820 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.356021 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.356112 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.358633 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.358634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.358634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.358641 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.360014 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.360514 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.360515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.360515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.360565 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.360823 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.360823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.360824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.360918 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.360996 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.361344 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.361415 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:22:43.361606 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.352649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46075 (local address 127.0.0.1:47540)
0924 07:22:43.352805 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.352845 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:43.353442 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:43.353774 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:43.353778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:43.354022 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:43.354219 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.354223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.354907 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.354908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.355696 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.355699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.356146 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.356147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.356273 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.356796 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:43.356804 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:43.358523 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:43.360101 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.360109 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.360111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.360437 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.360631 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.360632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.360633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.360761 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.360992 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:43.360999 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:43.361158 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 07:22:43.361299 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 07:22:43.361427 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260924 07:22:43.362282 26425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.124@KRBTEST.COM" created.
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.125@KRBTEST.COM for kudu/127.3.246.125@KRBTEST.COM
I20260924 07:22:43.370090 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.362517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:39679 (local address 127.0.0.1:60086)
0924 07:22:43.362563 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.362574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:43.362615 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:43.362962 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:43.362963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:43.363099 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:43.363141 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.363142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.364334 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.364335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.364886 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.364889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.365374 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.365375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.365436 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.365857 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:43.365862 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:43.366983 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:43.369040 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.369041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.369041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.369218 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.369435 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.369436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.369436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.369483 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.369845 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:43.369845 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:43.369880 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 07:22:43.369933 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:22:43.370006 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 07:22:43.370219 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.362597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.3.246.125:39679)
0924 07:22:43.362788 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.362794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.362815 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:22:43.362836 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.362838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.362876 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.362961 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.363245 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.364222 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.364985 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.365285 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.365374 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.367067 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.367076 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.367077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.367084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.368953 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.369301 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.369304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.369304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.369368 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.369542 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.369543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.369543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.369776 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.369871 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.369967 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.370087 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260924 07:22:43.370842 26425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:22:43.371181 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.371063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:44869 (local address 127.0.0.1:36474)
0924 07:22:43.371108 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.371157 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 07:22:43.371368 26374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
I20260924 07:22:43.371802 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.371710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:44869 (local address 127.0.0.1:36484)
0924 07:22:43.371759 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.371785 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:22:43.371961 26425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:44869: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.124.keytab.
Entry for principal kudu/127.3.246.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.124.keytab.
I20260924 07:22:43.378919 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.124:44869
--webserver_interface=127.3.246.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.124.keytab
--principal=kudu/127.3.246.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:42851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:46075,127.3.246.125:39679,127.3.246.124:44869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:22:43.421250 26425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } attempt: 1
I20260924 07:22:43.421761 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.421601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:44869 (local address 127.0.0.1:36494)
0924 07:22:43.421665 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.421731 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 07:22:43.422093 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.421996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:44869 (local address 127.0.0.1:36496)
0924 07:22:43.422040 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.422063 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 07:22:43.422178 26425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:44869: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
I20260924 07:22:43.423641 26350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } attempt: 3
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for kudu/127.3.246.125@KRBTEST.COM
I20260924 07:22:43.430910 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.424076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.3.246.125:39679)
0924 07:22:43.424135 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.424137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.424152 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:22:43.424176 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.424177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.424185 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.424233 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.424572 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.425064 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.426065 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.426275 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.426327 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.428312 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.428315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.428316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.428327 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.429811 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.430149 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.430150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.430151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.430211 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.430406 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.430406 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.430407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.430522 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.430593 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.430744 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.430820 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:22:43.430915 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.424009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:39679 (local address 127.0.0.1:60090)
0924 07:22:43.424046 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.424059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:43.424122 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:43.424243 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:43.424244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:43.424429 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:43.424504 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.424506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.425136 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.425137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.425992 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.425994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.426342 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.426343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.426415 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.426843 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:43.426848 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:43.428198 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:43.429889 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.429891 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.429891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.430065 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.430277 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.430278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.430279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.430339 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.430595 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:43.430597 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:43.430640 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 07:22:43.430719 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:22:43.430820 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 07:22:43.431492 26350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:22:43.431800 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.431702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:44869 (local address 127.0.0.1:36508)
0924 07:22:43.431748 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.431775 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 07:22:43.432121 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.432058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:44869 (local address 127.0.0.1:36524)
0924 07:22:43.432090 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.432109 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 07:22:43.432215 26350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:44869: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
W20260924 07:22:43.462234 26432 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:22:43.462426 26432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:43.462443 26432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:43.462462 26432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 07:22:43.463348 26350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } attempt: 1
W20260924 07:22:43.463541 26432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:43.463591 26432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:43.463606 26432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:43.463619 26432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:43.463660 26432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:43.463820 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.463685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:44869 (local address 127.0.0.1:36536)
0924 07:22:43.463739 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.463787 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 07:22:43.464183 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.464080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:44869 (local address 127.0.0.1:36540)
0924 07:22:43.464130 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.464160 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 07:22:43.464294 26350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:44869: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
I20260924 07:22:43.465337 26432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:42851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:46075,127.3.246.125:39679,127.3.246.124:44869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.124.keytab
--principal=kudu/127.3.246.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.124:44869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.246.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:43.465576 26432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:43.465826 26432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:43.470336 26440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:43.470391 26437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:43.470533 26432 server_base.cc:1034] running on GCE node
W20260924 07:22:43.470624 26438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:43.470862 26432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:43.471100 26432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:43.472239 26432 hybrid_clock.cc:648] HybridClock initialized: now 1790234563472218 us; error 40 us; skew 500 ppm
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:43.474686 26432 init.cc:377] Logged in from keytab as kudu/127.3.246.124@KRBTEST.COM (short username kudu)
I20260924 07:22:43.475138 26432 webserver.cc:460] Webserver started at http://127.3.246.124:36283/ using document root <none> and password file <none>
I20260924 07:22:43.475373 26432 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:43.475440 26432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:43.475589 26432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:43.476622 26432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/data/instance:
uuid: "60857526dd394247823d0b08dcb46dd8"
format_stamp: "Formatted at 2026-09-24 07:22:43 on dist-test-slave-rc1w"
server_key: "31fc7bcb77838cd04300162b997e026a"
server_key_iv: "954b9402473c1d29e6079f31b74a0b37"
server_key_version: "encryptionkey@0"
I20260924 07:22:43.476972 26432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/wal/instance:
uuid: "60857526dd394247823d0b08dcb46dd8"
format_stamp: "Formatted at 2026-09-24 07:22:43 on dist-test-slave-rc1w"
server_key: "31fc7bcb77838cd04300162b997e026a"
server_key_iv: "954b9402473c1d29e6079f31b74a0b37"
server_key_version: "encryptionkey@0"
I20260924 07:22:43.478332 26432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:22:43.479007 26447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:43.479169 26432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:22:43.479276 26432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/wal
uuid: "60857526dd394247823d0b08dcb46dd8"
format_stamp: "Formatted at 2026-09-24 07:22:43 on dist-test-slave-rc1w"
server_key: "31fc7bcb77838cd04300162b997e026a"
server_key_iv: "954b9402473c1d29e6079f31b74a0b37"
server_key_version: "encryptionkey@0"
I20260924 07:22:43.479347 26432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:43.490312 26425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } attempt: 2
I20260924 07:22:43.490684 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.490569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:44869 (local address 127.0.0.1:36550)
0924 07:22:43.490623 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.490658 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 07:22:43.491097 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.490982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:44869 (local address 127.0.0.1:36566)
0924 07:22:43.491036 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.491067 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 07:22:43.491207 26425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:44869: Network error: Client connection negotiation failed: client connection to 127.3.246.124:44869: connect: Connection refused (error 111)
I20260924 07:22:43.492159 26432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:43.494009 26432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:43.494158 26432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:43.498209 26432 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.124:44869
I20260924 07:22:43.498239 26499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.124:44869 every 8 connection(s)
I20260924 07:22:43.498574 26432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/data/info.pb
I20260924 07:22:43.499475 26500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:43.500895 26500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:22:43.503291 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 26432
I20260924 07:22:43.503420 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234301618289-4057-0/minicluster-data/master-2/wal/instance
I20260924 07:22:43.503589 4057 external_mini_cluster.cc:1428] Setting key 1bd651e15da9a6fa692a3c01b3542840
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.124@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.124@KRBTEST.COM
I20260924 07:22:43.510335 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.501506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47554 (local address 127.3.246.126:46075)
0924 07:22:43.501565 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.501566 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.501754 (+ 188us) server_negotiation.cc:407] Connection header received
0924 07:22:43.502194 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.502195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.502203 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.502263 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.502804 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.503265 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.504781 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.504988 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.505057 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.507507 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.507508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.507509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.507515 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.508947 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.509357 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.509358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.509359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.509421 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.509665 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.509666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.509667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.509810 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.509886 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.510149 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.510242 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 07:22:43.510535 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.501468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46075 (local address 127.0.0.1:47554)
0924 07:22:43.501619 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.501658 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:43.502115 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:43.502323 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:43.502326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:43.502554 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:43.502736 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.502739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.503348 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.503350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.504683 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.504686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.505090 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.505091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.505201 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.505795 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:43.505804 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:43.507417 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:43.509047 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.509053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.509055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.509278 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.509489 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.509490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.509490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.509593 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.509890 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:43.509897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:43.510023 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 07:22:43.510128 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 07:22:43.510303 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260924 07:22:43.511114 26500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:22:43.512679 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.503948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.3.246.124:44869)
0924 07:22:43.504080 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.504083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.504147 (+ 64us) server_negotiation.cc:407] Connection header received
0924 07:22:43.504224 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.504226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.504241 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.504300 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.504997 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.505908 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.506825 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.507222 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.507278 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.509709 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.509717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.509717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.509730 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.511498 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.511850 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.511853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.511853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.511913 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.512122 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.512123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.512124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.512260 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.512390 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.512483 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.512563 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.124@KRBTEST.COM for kudu/127.3.246.125@KRBTEST.COM
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:43.518146 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.511377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:39679 (local address 127.0.0.1:60106)
0924 07:22:43.511423 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.511433 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:43.511469 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:43.511635 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:43.511636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:43.511766 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:43.511791 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.511792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.512385 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.512386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.513017 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.513019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.513486 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.513488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.513560 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.513992 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:43.513997 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:43.515330 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:43.517094 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.517094 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.517095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.517249 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.517467 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.517468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.517468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.517532 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.517838 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:43.517839 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:43.517879 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 07:22:43.517965 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 07:22:43.518054 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 07:22:43.518151 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.511457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.3.246.125:39679)
0924 07:22:43.511510 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.511512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.511520 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:22:43.511543 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.511544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.511553 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.511619 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.511864 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.512309 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.513098 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.513423 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.513491 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.515417 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.515418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.515419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.515426 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.517014 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.517334 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.517335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.517335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.517390 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.517598 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.517599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.517599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.517764 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.517845 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.517990 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.518066 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:22:43.518535 26500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:22:43.520841 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.513402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.3.246.126:46075)
0924 07:22:43.513435 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.513437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.513445 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:22:43.513506 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.513506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.513515 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.513570 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.514043 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.514673 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.515472 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.515691 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.515773 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.518090 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.518091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.518092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.518098 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.519769 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.520103 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.520104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.520104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.520172 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.520360 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.520361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.520361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.520484 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.520650 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.520685 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.520753 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.124@KRBTEST.COM for kudu/127.3.246.124@KRBTEST.COM
I20260924 07:22:43.525851 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.518783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:44869 (local address 127.0.0.1:36594)
0924 07:22:43.518826 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.518835 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:43.518869 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:43.519268 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:43.519269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:43.519447 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:43.519478 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.519479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.520189 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.520190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.520924 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.520926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.521478 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.521479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.521541 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.521898 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:43.521904 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:43.523038 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:43.524727 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.524728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.524729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.524887 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.525090 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.525091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.525091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.525139 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.525438 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:43.525439 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:43.525485 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 07:22:43.525572 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:22:43.525662 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 07:22:43.525851 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.518842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36594 (local address 127.3.246.124:44869)
0924 07:22:43.518873 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.518875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.518882 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:22:43.519188 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.519189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.519197 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.519246 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.519552 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.520120 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.521006 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.521259 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.521320 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.523133 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.523134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.523135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.523140 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.524635 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.524962 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.524964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.524964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.525022 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.525198 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.525198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.525199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.525340 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.525405 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.525597 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.525754 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":16}
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.125@KRBTEST.COM
I20260924 07:22:43.527601 26500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8: Bootstrap starting.
I20260924 07:22:43.528463 26500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:43.528810 26500 log.cc:824] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:43.528981 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.521431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.3.246.125:39679)
0924 07:22:43.521462 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.521464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.521499 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:22:43.521564 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.521564 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.521571 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.521605 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.522206 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.522613 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.523579 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.523871 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.523938 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.526505 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.526506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.526507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.526513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.527851 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.528176 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.528177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.528177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.528228 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.528430 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.528431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.528431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.528540 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.528705 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.528807 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.528861 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 07:22:43.529939 26500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8: No bootstrap required, opened a new log
I20260924 07:22:43.530473 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:43.531637 26500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } }
I20260924 07:22:43.531785 26500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60857526dd394247823d0b08dcb46dd8, State: Initialized, Role: FOLLOWER
I20260924 07:22:43.531876 26500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } }
I20260924 07:22:43.532128 26506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } } }
I20260924 07:22:43.532182 26500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:43.531455 26324 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0beb1b85d44248f89e7eaf8ecdad420c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47570
I20260924 07:22:43.532236 26506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:22:43.533913 26517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:22:43.533977 26517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:22:43.534171 26432 master_runner.cc:186] Error getting master registration for 127.3.246.126:46075: OK, instance_id { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" instance_seqno: 1790234563197609 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:22:43.536970 26432 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:22:43.539438 26350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } attempt: 2
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for kudu/127.3.246.124@KRBTEST.COM
I20260924 07:22:43.547307 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.539674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:44869 (local address 127.0.0.1:36596)
0924 07:22:43.539727 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.539739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:43.539777 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:43.540026 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:43.540027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:43.540195 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:43.540241 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.540242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.541105 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.541105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.541916 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.541918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.542403 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.542403 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.542455 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.542909 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:43.542914 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:43.544411 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:43.546193 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.546194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.546194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.546460 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.546665 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.546666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.546666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.546727 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.547005 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:43.547005 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:43.547055 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 07:22:43.547121 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:22:43.547205 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 07:22:43.547325 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.539851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36596 (local address 127.3.246.124:44869)
0924 07:22:43.539910 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.539912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.539920 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:22:43.539941 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.539942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.539950 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.540014 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.540320 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.541020 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.541994 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.542313 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.542406 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.544476 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.544478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.544478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.544485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.546099 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.546540 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.546541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.546542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.546599 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.546789 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.546789 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.546790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.546940 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.547009 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.547159 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.547239 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:22:43.548977 26350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c: Bootstrap starting.
I20260924 07:22:43.549707 26350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:43.550019 26350 log.cc:824] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:43.550839 26350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c: No bootstrap required, opened a new log
I20260924 07:22:43.552237 26350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } }
I20260924 07:22:43.552376 26350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0beb1b85d44248f89e7eaf8ecdad420c, State: Initialized, Role: FOLLOWER
I20260924 07:22:43.552457 26350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } }
I20260924 07:22:43.552762 26350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:43.552717 26520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } } }
I20260924 07:22:43.552791 26324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 07:22:43.552829 26520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:22:43.552847 26324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:43.553550 26324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } }
I20260924 07:22:43.554267 26474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "60857526dd394247823d0b08dcb46dd8"
I20260924 07:22:43.554390 26474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8 [term 0 FOLLOWER]: Advancing to term 1
W20260924 07:22:43.554559 26531 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:22:43.554610 26531 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:22:43.555308 26474 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0beb1b85d44248f89e7eaf8ecdad420c in term 1.
I20260924 07:22:43.555461 26324 leader_election.cc:288] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [CANDIDATE]: Term 1 election: Requested vote from peers 137179ffac554858934fae019c6d01d8 (127.3.246.125:39679), 60857526dd394247823d0b08dcb46dd8 (127.3.246.124:44869)
I20260924 07:22:43.555493 26299 leader_election.cc:302] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0beb1b85d44248f89e7eaf8ecdad420c, 60857526dd394247823d0b08dcb46dd8; no voters:
I20260924 07:22:43.555629 26520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:43.555795 26520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [term 1 LEADER]: Becoming Leader. State: Replica: 0beb1b85d44248f89e7eaf8ecdad420c, State: Running, Role: LEADER
I20260924 07:22:43.555886 26520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } }
I20260924 07:22:43.556294 26520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0beb1b85d44248f89e7eaf8ecdad420c. Latest consensus state: current_term: 1 leader_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } } }
I20260924 07:22:43.556358 26520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:43.556628 26282 master_runner.cc:186] Error getting master registration for 127.3.246.125:39679: OK, instance_id { permanent_uuid: "137179ffac554858934fae019c6d01d8" instance_seqno: 1790234563348024 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:22:43.558959 26282 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:22:43.559520 26539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:43.559721 26539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:43.561736 26474 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8 [term 1 FOLLOWER]: Refusing update from remote peer 0beb1b85d44248f89e7eaf8ecdad420c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:43.562103 26533 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:43.563622 26506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0beb1b85d44248f89e7eaf8ecdad420c. Latest consensus state: current_term: 1 leader_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } } }
I20260924 07:22:43.563726 26506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:22:43.565629 26506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } } }
I20260924 07:22:43.565738 26506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:22:43.564920 26540 mvcc.cc:205] Tried to move back new op lower bound from 7332800772345163776 to 7332800772325568512. Current Snapshot: MvccSnapshot[applied={T|T < 7332800772345163776}]
I20260924 07:22:43.565972 26520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } } }
I20260924 07:22:43.566046 26520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:43.567104 26544 mvcc.cc:205] Tried to move back new op lower bound from 7332800772345163776 to 7332800772325568512. Current Snapshot: MvccSnapshot[applied={T|T < 7332800772345163776}]
I20260924 07:22:43.567952 26539 catalog_manager.cc:1347] Generated new cluster ID: 16da86c631ad4c33b05521ae9383a19c
I20260924 07:22:43.568001 26539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:43.575685 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.560628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.3.246.125:39679)
0924 07:22:43.560674 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.560676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.561861 (+ 1185us) server_negotiation.cc:407] Connection header received
0924 07:22:43.562438 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.562440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.562451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.562482 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.563729 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.564465 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.567274 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.567580 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.567630 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.570801 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.570802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.570803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.570809 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.572330 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.574617 (+ 2287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.574618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.574619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.574675 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.574859 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.574860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.574860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.574982 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.575070 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.575513 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.575591 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:22:43.577016 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.560854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.3.246.126:46075)
0924 07:22:43.560899 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.560901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.561760 (+ 859us) server_negotiation.cc:407] Connection header received
0924 07:22:43.562018 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.562019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.562043 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.562076 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.565529 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.566210 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.568231 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.568509 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.568544 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.572679 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.572680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.572681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.572688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.574210 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.575238 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.575239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.575239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.575297 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.576448 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.576449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.576449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.576580 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.576646 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.576859 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.576930 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:22:43.580713 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.561330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.3.246.124:44869)
0924 07:22:43.561442 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.561444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.573540 (+ 12096us) server_negotiation.cc:407] Connection header received
0924 07:22:43.573644 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.573645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.573656 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.573712 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.574369 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.575106 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.576139 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.576424 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.576466 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.577935 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.577937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.577937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.577943 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.579456 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.579850 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.579850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.579851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.579916 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.580107 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.580108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.580108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.580237 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.580340 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.580545 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.580625 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 07:22:43.582337 26425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } attempt: 3
Sep 24 07:22:43 dist-test-slave-rc1w krb5kdc[26269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.125@KRBTEST.COM for kudu/127.3.246.124@KRBTEST.COM
I20260924 07:22:43.589753 26539 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:43.591326 26539 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:43.591363 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.582569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:44869 (local address 127.0.0.1:36612)
0924 07:22:43.582614 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:43.582625 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:43.582664 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:43.582875 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:43.582876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:43.583025 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:43.583070 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.583071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.583934 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.583934 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.584687 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:43.584689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.585906 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.585907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:43.585982 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.586403 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:43.586420 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:43.587838 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:43.590269 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.590271 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.590271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.590475 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.590673 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:43.590673 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:43.590674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:43.590729 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:43.591068 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:43.591068 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:43.591111 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 07:22:43.591157 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 07:22:43.591256 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 07:22:43.591856 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:43.582687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.3.246.124:44869)
0924 07:22:43.582759 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.582761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.582767 (+ 6us) server_negotiation.cc:407] Connection header received
0924 07:22:43.582796 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.582797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.582806 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.582857 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.583162 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.583846 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.584782 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:43.585079 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:43.585131 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:43.587915 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:43.587916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:43.587917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:43.587924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:43.590175 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.590554 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.590555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.590555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.590606 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:43.590799 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:43.590800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:43.590800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:43.590973 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:43.591657 (+ 684us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:43.591698 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 07:22:43.591770 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:22:43.593392 26425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8: Bootstrap starting.
I20260924 07:22:43.594194 26425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:43.594499 26425 log.cc:824] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:43.600646 26425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8: No bootstrap required, opened a new log
I20260924 07:22:43.601748 26539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c: Generated new TSK 0
I20260924 07:22:43.601938 26539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:22:43.603615 26425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } }
I20260924 07:22:43.603750 26425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137179ffac554858934fae019c6d01d8, State: Initialized, Role: FOLLOWER
I20260924 07:22:43.603904 26425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } }
I20260924 07:22:43.604272 26550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } } }
I20260924 07:22:43.604380 26398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:43.604522 26550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:22:43.605232 26398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8 [term 1 FOLLOWER]: Refusing update from remote peer 0beb1b85d44248f89e7eaf8ecdad420c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:43.604321 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "137179ffac554858934fae019c6d01d8"
I20260924 07:22:43.605518 26547 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [LEADER]: Connected to new peer: Peer: permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:43.606153 26399 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0beb1b85d44248f89e7eaf8ecdad420c in term 1.
I20260924 07:22:43.604290 26425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:43.607291 26550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0beb1b85d44248f89e7eaf8ecdad420c. Latest consensus state: current_term: 1 leader_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } } }
I20260924 07:22:43.607419 26550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:22:43.607486 26550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } } }
I20260924 07:22:43.607565 26550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:22:43.608840 26564 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:22:43.608923 26564 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:22:43.609632 26551 mvcc.cc:205] Tried to move back new op lower bound from 7332800772506693632 to 7332800772325568512. Current Snapshot: MvccSnapshot[applied={T|T < 7332800772345163776}]
W20260924 07:22:43.629576 26353 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:43.627960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.3.246.126:46075)
0924 07:22:43.628004 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.628006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.628902 (+ 896us) server_negotiation.cc:407] Connection header received
0924 07:22:43.629009 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.629011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.629022 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.629066 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.629503 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47598: BlockingRecv error: recv got EOF from 127.0.0.1:47598 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 07:22:43.630523 26429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:43.628365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.3.246.125:39679)
0924 07:22:43.628404 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.628405 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.629990 (+ 1585us) server_negotiation.cc:407] Connection header received
0924 07:22:43.630102 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.630103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.630114 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.630143 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.630458 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60128: BlockingRecv error: recv got EOF from 127.0.0.1:60128 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 07:22:43.631349 26504 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:43.628654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.3.246.124:44869)
0924 07:22:43.628692 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.628693 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.630832 (+ 2139us) server_negotiation.cc:407] Connection header received
0924 07:22:43.630957 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.630958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.630968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.630996 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.631291 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36624: BlockingRecv error: recv got EOF from 127.0.0.1:36624 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 07:22:43.633455 26429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:43.632732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.3.246.125:39679)
0924 07:22:43.632764 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.632766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.632947 (+ 181us) server_negotiation.cc:407] Connection header received
0924 07:22:43.633062 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.633063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.633086 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.633116 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.633411 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60142: BlockingRecv error: recv got EOF from 127.0.0.1:60142 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 07:22:43.636859 26429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:43.636132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.3.246.125:39679)
0924 07:22:43.636164 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.636166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.636388 (+ 222us) server_negotiation.cc:407] Connection header received
0924 07:22:43.636481 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.636482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.636491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.636519 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.636815 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60158: BlockingRecv error: recv got EOF from 127.0.0.1:60158 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 07:22:43.640708 26429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:43.640001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60170 (local address 127.3.246.125:39679)
0924 07:22:43.640045 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.640047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.640257 (+ 210us) server_negotiation.cc:407] Connection header received
0924 07:22:43.640346 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.640347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.640356 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.640383 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.640667 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60170: BlockingRecv error: recv got EOF from 127.0.0.1:60170 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 07:22:43.646602 26429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:43.645884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.3.246.125:39679)
0924 07:22:43.645916 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.645917 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.646134 (+ 217us) server_negotiation.cc:407] Connection header received
0924 07:22:43.646226 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.646227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.646236 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.646264 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.646557 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60180: BlockingRecv error: recv got EOF from 127.0.0.1:60180 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 07:22:43.658118 26429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:43.657236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.3.246.125:39679)
0924 07:22:43.657274 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.657276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.657524 (+ 248us) server_negotiation.cc:407] Connection header received
0924 07:22:43.657630 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.657631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.657641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.657687 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.658049 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60190: BlockingRecv error: recv got EOF from 127.0.0.1:60190 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 07:22:43.676445 26429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:43.675529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.3.246.125:39679)
0924 07:22:43.675574 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.675576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.675857 (+ 281us) server_negotiation.cc:407] Connection header received
0924 07:22:43.675965 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.675966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.675977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.676008 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.676385 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60196: BlockingRecv error: recv got EOF from 127.0.0.1:60196 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 07:22:43.711035 26429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:43.710072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.3.246.125:39679)
0924 07:22:43.710119 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.710121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.710401 (+ 280us) server_negotiation.cc:407] Connection header received
0924 07:22:43.710511 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.710513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.710525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.710556 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.710974 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60204: BlockingRecv error: recv got EOF from 127.0.0.1:60204 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 07:22:43.777392 26429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:43.776665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.3.246.125:39679)
0924 07:22:43.776734 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.776736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.776840 (+ 104us) server_negotiation.cc:407] Connection header received
0924 07:22:43.776904 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.776905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.776916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.776958 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.777329 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60218: BlockingRecv error: recv got EOF from 127.0.0.1:60218 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 07:22:43.907797 26429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:43.906990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.3.246.125:39679)
0924 07:22:43.907124 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:43.907126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:43.907190 (+ 64us) server_negotiation.cc:407] Connection header received
0924 07:22:43.907261 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:43.907262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:43.907273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:43.907339 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:43.907727 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60228: BlockingRecv error: recv got EOF from 127.0.0.1:60228 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 07:22:44.160506 26547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0beb1b85d44248f89e7eaf8ecdad420c" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46075 } } peers { permanent_uuid: "137179ffac554858934fae019c6d01d8" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 39679 } } peers { permanent_uuid: "60857526dd394247823d0b08dcb46dd8" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 44869 } } }
I20260924 07:22:44.160681 26547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0beb1b85d44248f89e7eaf8ecdad420c [sys.catalog]: This master's current role is: LEADER
W20260924 07:22:44.165843 26429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:44.165233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.3.246.125:39679)
0924 07:22:44.165303 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:44.165305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:44.165313 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:22:44.165347 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:44.165348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:44.165357 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:44.165427 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:44.165791 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60242: BlockingRecv error: recv got EOF from 127.0.0.1:60242 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:22:44.534395 26517 catalog_manager.cc:1259] Loaded cluster ID: 16da86c631ad4c33b05521ae9383a19c
I20260924 07:22:44.534476 26517 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8: loading cluster ID for follower catalog manager: success
I20260924 07:22:44.535280 26517 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8: acquiring CA information for follower catalog manager: success
I20260924 07:22:44.535620 26517 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60857526dd394247823d0b08dcb46dd8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:22:44.609354 26564 catalog_manager.cc:1259] Loaded cluster ID: 16da86c631ad4c33b05521ae9383a19c
I20260924 07:22:44.609437 26564 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8: loading cluster ID for follower catalog manager: success
I20260924 07:22:44.610420 26564 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8: acquiring CA information for follower catalog manager: success
I20260924 07:22:44.610747 26564 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 137179ffac554858934fae019c6d01d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:22:44.681887 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:44.679132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.3.246.125:39679)
0924 07:22:44.679250 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:44.679253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:44.679266 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:22:44.679375 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:44.679376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:44.679436 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:44.679524 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:44.679690 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:44.680241 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:44.681035 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:44.681249 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:44.681324 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:44.681518 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:44.681627 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:44.681747 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:44.681770 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 07:22:44.681791 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":50,"threads_started":1}
I20260924 07:22:44.685807 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:44.683683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.3.246.124:44869)
0924 07:22:44.683814 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:44.683815 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:44.683823 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:22:44.683856 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:44.683857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:44.683887 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:44.683950 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:44.684029 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:44.684482 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:44.685137 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:44.685318 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:44.685365 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:44.685488 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:44.685575 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:44.685625 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:44.685717 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 07:22:44.685738 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260924 07:22:44.690263 26647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:44.687422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47600 (local address 127.3.246.126:46075)
0924 07:22:44.687569 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:44.687572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:44.687699 (+ 127us) server_negotiation.cc:407] Connection header received
0924 07:22:44.687758 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:44.687759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:44.687812 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:44.687886 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:44.688049 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:44.688540 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:44.689284 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:44.689496 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:44.689572 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:44.689787 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:44.689882 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:44.689941 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:44.690137 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 07:22:44.690173 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":47,"threads_started":1}
I20260924 07:22:44.690511 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:44.687612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.3.246.124:44869)
0924 07:22:44.687646 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:44.687648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:44.687655 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:22:44.687710 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:44.687710 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:44.687720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:44.687771 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:44.687875 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:44.688564 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:44.689713 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:44.689985 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:44.690042 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:44.690227 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:44.690260 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:44.690311 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:44.690399 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 07:22:44.690419 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:22:44.690866 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:44.687418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.3.246.125:39679)
0924 07:22:44.687455 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:44.687457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:44.687576 (+ 119us) server_negotiation.cc:407] Connection header received
0924 07:22:44.687603 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:44.687604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:44.687613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:44.687662 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:44.687831 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:44.688561 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:44.689727 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:44.689943 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:44.690011 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:44.690156 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:44.690192 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:44.690237 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:44.690614 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 07:22:44.690639 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:22:44.692116 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 26282
I20260924 07:22:44.697389 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 26357
I20260924 07:22:44.702243 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 26432
2026-09-24T07:22:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2769 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:22:44 dist-test-slave-rc1w krb5kdc[26652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:22:44 dist-test-slave-rc1w krb5kdc[26652](info): set up 2 sockets
Sep 24 07:22:44 dist-test-slave-rc1w krb5kdc[26652](info): commencing operation
krb5kdc: starting...
W20260924 07:22:45.737994 4057 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/test-user.keytab.
Sep 24 07:22:45 dist-test-slave-rc1w krb5kdc[26652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:22:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:22:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
Entry for principal kudu/127.3.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab.
I20260924 07:22:45.854640 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:46591
--webserver_interface=127.3.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:46591,127.3.246.125:46779,127.3.246.124:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:22:45.933329 26665 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:22:45.933578 26665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:45.933609 26665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:45.933651 26665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:45.934778 26665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:45.934847 26665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:45.934871 26665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:45.934899 26665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:45.934929 26665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:45.936401 26665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:46591,127.3.246.125:46779,127.3.246.124:34473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.126.keytab
--principal=kudu/127.3.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.126:46591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:45.936669 26665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:45.936887 26665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:22:45.941663 26673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:45.941725 26670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:45.941838 26665 server_base.cc:1034] running on GCE node
W20260924 07:22:45.941735 26671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:45.942178 26665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:45.942433 26665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:45.943624 26665 hybrid_clock.cc:648] HybridClock initialized: now 1790234565943613 us; error 47 us; skew 500 ppm
Sep 24 07:22:45 dist-test-slave-rc1w krb5kdc[26652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:45.954288 26665 init.cc:377] Logged in from keytab as kudu/127.3.246.126@KRBTEST.COM (short username kudu)
I20260924 07:22:45.954780 26665 webserver.cc:460] Webserver started at http://127.3.246.126:38445/ using document root <none> and password file <none>
I20260924 07:22:45.955015 26665 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:45.955084 26665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:45.955240 26665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:45.956311 26665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/data/instance:
uuid: "287323bc23544c0297e260e285541a2b"
format_stamp: "Formatted at 2026-09-24 07:22:45 on dist-test-slave-rc1w"
server_key: "0b356d1bbf4d713150c30d9f1a641ca7"
server_key_iv: "804b1467bd989aa149b896fe4f4e5ce4"
server_key_version: "encryptionkey@0"
I20260924 07:22:45.956674 26665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/wal/instance:
uuid: "287323bc23544c0297e260e285541a2b"
format_stamp: "Formatted at 2026-09-24 07:22:45 on dist-test-slave-rc1w"
server_key: "0b356d1bbf4d713150c30d9f1a641ca7"
server_key_iv: "804b1467bd989aa149b896fe4f4e5ce4"
server_key_version: "encryptionkey@0"
I20260924 07:22:45.958043 26665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:22:45.958743 26680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:45.958909 26665 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:22:45.959007 26665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/wal
uuid: "287323bc23544c0297e260e285541a2b"
format_stamp: "Formatted at 2026-09-24 07:22:45 on dist-test-slave-rc1w"
server_key: "0b356d1bbf4d713150c30d9f1a641ca7"
server_key_iv: "804b1467bd989aa149b896fe4f4e5ce4"
server_key_version: "encryptionkey@0"
I20260924 07:22:45.959085 26665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:45.968802 26665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:45.971822 26665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:45.971977 26665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:45.976027 26665 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.126:46591
I20260924 07:22:45.976056 26732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.126:46591 every 8 connection(s)
I20260924 07:22:45.976397 26665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/data/info.pb
I20260924 07:22:45.977342 26733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:45.978902 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 26665
I20260924 07:22:45.979027 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-0/wal/instance
I20260924 07:22:45.978968 26733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:22:45.979219 4057 external_mini_cluster.cc:1428] Setting key 211f473195675b1b7ae927b5304e368d
Sep 24 07:22:45 dist-test-slave-rc1w krb5kdc[26652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.246.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.125@KRBTEST.COM" created.
I20260924 07:22:45.990878 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:45.979791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48054 (local address 127.3.246.126:46591)
0924 07:22:45.979993 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:45.979998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:45.980053 (+ 55us) server_negotiation.cc:407] Connection header received
0924 07:22:45.980599 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:45.980601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:45.980630 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:45.980733 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:45.981264 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:45.982201 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:45.983169 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:45.983482 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:45.983596 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:45.986404 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:45.986410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:45.986411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:45.986417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:45.988909 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:45.989408 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:45.989429 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:45.989429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:45.989546 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:45.989846 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:45.989848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:45.989848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:45.990178 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:45.990269 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:45.990468 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 07:22:45.990665 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
I20260924 07:22:45.990886 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:45.979751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46591 (local address 127.0.0.1:48054)
0924 07:22:45.979920 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:45.979967 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:45.980434 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:45.980739 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:45.980741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:45.981003 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:45.981152 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:45.981157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:45.982313 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:45.982314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:45.983044 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:45.983046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:45.983593 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:45.983594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:45.983658 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:45.984905 (+ 1247us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:45.984914 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:45.986303 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:45.989034 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:45.989041 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:45.989043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:45.989319 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:45.989633 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:45.989634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:45.989634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:45.989776 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:45.990271 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:45.990278 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:45.990336 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 07:22:45.990414 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:22:45.990570 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260924 07:22:45.991688 26733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:22:45.992091 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:45.991952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:46779 (local address 127.0.0.1:51044)
0924 07:22:45.992011 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:45.992063 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 07:22:45.992257 26683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.246.125:46779: connect: Connection refused (error 111)
I20260924 07:22:45.992672 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:45.992554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:46779 (local address 127.0.0.1:51050)
0924 07:22:45.992595 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:45.992646 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 07:22:45.992815 26733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.125:46779: Network error: Client connection negotiation failed: client connection to 127.3.246.125:46779: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.125.keytab.
Entry for principal kudu/127.3.246.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.125.keytab.
I20260924 07:22:46.005012 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.125:46779
--webserver_interface=127.3.246.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.125.keytab
--principal=kudu/127.3.246.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:46591,127.3.246.125:46779,127.3.246.124:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:22:46.042109 26733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } attempt: 1
I20260924 07:22:46.042629 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.042467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:46779 (local address 127.0.0.1:51052)
0924 07:22:46.042537 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.042587 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 07:22:46.043020 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.042928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:46779 (local address 127.0.0.1:51062)
0924 07:22:46.042971 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.043000 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 07:22:46.043119 26733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.125:46779: Network error: Client connection negotiation failed: client connection to 127.3.246.125:46779: connect: Connection refused (error 111)
W20260924 07:22:46.084882 26740 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:22:46.085088 26740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:46.085105 26740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:46.085125 26740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:46.086305 26740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:46.086345 26740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:46.086360 26740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:46.086373 26740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:46.086385 26740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:46.087862 26740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:46591,127.3.246.125:46779,127.3.246.124:34473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.125.keytab
--principal=kudu/127.3.246.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.125:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.246.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:46.088106 26740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:46.088330 26740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:22:46.093204 26740 server_base.cc:1034] running on GCE node
W20260924 07:22:46.093097 26746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:46.093079 26745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:46.093082 26748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:46.093575 26740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:46.093884 26740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:46.095031 26740 hybrid_clock.cc:648] HybridClock initialized: now 1790234566095024 us; error 29 us; skew 500 ppm
Sep 24 07:22:46 dist-test-slave-rc1w krb5kdc[26652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:46.097950 26740 init.cc:377] Logged in from keytab as kudu/127.3.246.125@KRBTEST.COM (short username kudu)
I20260924 07:22:46.098377 26740 webserver.cc:460] Webserver started at http://127.3.246.125:41253/ using document root <none> and password file <none>
I20260924 07:22:46.098598 26740 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:46.098664 26740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:46.098819 26740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:46.099900 26740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/data/instance:
uuid: "c6e32a14af484c5bb1e2ab3933b85b25"
format_stamp: "Formatted at 2026-09-24 07:22:46 on dist-test-slave-rc1w"
server_key: "d12100e1c99f8441a54fdafe54b963b5"
server_key_iv: "0a8a20512e867de4951653c639a541bf"
server_key_version: "encryptionkey@0"
I20260924 07:22:46.100266 26740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/wal/instance:
uuid: "c6e32a14af484c5bb1e2ab3933b85b25"
format_stamp: "Formatted at 2026-09-24 07:22:46 on dist-test-slave-rc1w"
server_key: "d12100e1c99f8441a54fdafe54b963b5"
server_key_iv: "0a8a20512e867de4951653c639a541bf"
server_key_version: "encryptionkey@0"
I20260924 07:22:46.101574 26740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 07:22:46.102331 26755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:46.102496 26740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:22:46.102593 26740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/wal
uuid: "c6e32a14af484c5bb1e2ab3933b85b25"
format_stamp: "Formatted at 2026-09-24 07:22:46 on dist-test-slave-rc1w"
server_key: "d12100e1c99f8441a54fdafe54b963b5"
server_key_iv: "0a8a20512e867de4951653c639a541bf"
server_key_version: "encryptionkey@0"
I20260924 07:22:46.102674 26740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:46.111260 26733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } attempt: 2
I20260924 07:22:46.111756 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.111577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:46779 (local address 127.0.0.1:51070)
0924 07:22:46.111685 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.111726 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 07:22:46.112160 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.112056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:46779 (local address 127.0.0.1:51086)
0924 07:22:46.112106 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.112138 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.125:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 07:22:46.112241 26733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.125:46779: Network error: Client connection negotiation failed: client connection to 127.3.246.125:46779: connect: Connection refused (error 111)
I20260924 07:22:46.118350 26740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:46.121639 26740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:46.121869 26740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:46.125705 26740 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.125:46779
I20260924 07:22:46.125738 26807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.125:46779 every 8 connection(s)
I20260924 07:22:46.126104 26740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/data/info.pb
I20260924 07:22:46.127280 26808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:46.128998 26808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:22:46.129498 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 26740
I20260924 07:22:46.129614 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-1/wal/instance
I20260924 07:22:46.129856 4057 external_mini_cluster.cc:1428] Setting key fb0b2acbe3b5ae6b8f65f0d47e93499f
Sep 24 07:22:46 dist-test-slave-rc1w krb5kdc[26652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.125@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.246.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.246.124@KRBTEST.COM" created.
I20260924 07:22:46.140588 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.130060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.3.246.126:46591)
0924 07:22:46.130137 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.130140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.130150 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:22:46.130615 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.130617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.130633 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.130707 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.131441 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.131938 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.133017 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.133391 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.133490 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.136397 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.136399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.136399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.136408 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.139069 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.139505 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.139506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.139506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.139553 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.139813 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.139814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.139814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.139956 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.140025 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.140403 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.140496 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:22:46.140789 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.129753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46591 (local address 127.0.0.1:48068)
0924 07:22:46.129951 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.130019 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:46.130506 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:46.130762 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:46.130765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:46.131008 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:46.131324 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.131330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.132037 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.132039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.132910 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.132912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.133500 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.133501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.133617 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.134960 (+ 1343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:46.134968 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:46.136293 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:46.139147 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.139155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.139157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.139443 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.139608 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.139608 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.139609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.139718 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.140058 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:46.140065 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:46.140239 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 07:22:46.140362 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 07:22:46.140528 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260924 07:22:46.141283 26808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:22:46 dist-test-slave-rc1w krb5kdc[26652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.125@KRBTEST.COM for kudu/127.3.246.125@KRBTEST.COM
I20260924 07:22:46.150913 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.141565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:46779 (local address 127.0.0.1:51092)
0924 07:22:46.141613 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.141624 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:46.141666 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:46.141964 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:46.141964 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:46.142074 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:46.142125 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.142126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.143280 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.143281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.144160 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.144163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.144693 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.144694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.144781 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.145654 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:46.145660 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:46.146968 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:46.149852 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.149853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.149853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.150014 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.150240 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.150241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.150241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.150296 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.150582 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:46.150583 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:46.150619 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 07:22:46.150707 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:22:46.150801 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 07:22:46.151096 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.141623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.3.246.125:46779)
0924 07:22:46.141793 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.141799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.141825 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:22:46.141853 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.141856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.141871 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.141962 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.142193 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.143176 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.144296 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.144606 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.144675 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.147067 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.147074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.147075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.147082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.149763 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.150107 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.150110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.150111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.150167 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.150368 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.150370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.150370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.150509 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.150603 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.150826 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.150894 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260924 07:22:46.151686 26808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:22:46.152088 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.151924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:34473 (local address 127.0.0.1:42942)
0924 07:22:46.151966 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.152062 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 07:22:46.152248 26758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
I20260924 07:22:46.152653 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.152560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:34473 (local address 127.0.0.1:42950)
0924 07:22:46.152606 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.152633 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 07:22:46.152786 26808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:34473: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.246.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.124.keytab.
Entry for principal kudu/127.3.246.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.124.keytab.
I20260924 07:22:46.156477 4057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwu82JT/build/release/bin/kudu
/tmp/dist-test-taskwu82JT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.124:34473
--webserver_interface=127.3.246.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.124.keytab
--principal=kudu/127.3.246.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.246.84:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.246.126:46591,127.3.246.125:46779,127.3.246.124:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:22:46.202067 26808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } attempt: 1
I20260924 07:22:46.202600 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.202455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:34473 (local address 127.0.0.1:42956)
0924 07:22:46.202525 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.202566 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 07:22:46.202962 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.202861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:34473 (local address 127.0.0.1:42960)
0924 07:22:46.202910 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.202936 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 07:22:46.203056 26808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:34473: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
I20260924 07:22:46.203374 26733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } attempt: 3
Sep 24 07:22:46 dist-test-slave-rc1w krb5kdc[26652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for kudu/127.3.246.125@KRBTEST.COM
I20260924 07:22:46.212601 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.203667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:46779 (local address 127.0.0.1:51104)
0924 07:22:46.203754 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.203767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:46.203845 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:46.204106 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:46.204108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:46.204315 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:46.204382 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.204384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.205080 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.205082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.205665 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.205713 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.206139 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.206140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.206225 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.207414 (+ 1189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:46.207419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:46.208810 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:46.211511 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.211512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.211512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.211703 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.211923 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.211924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.211925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.211986 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.212254 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:46.212255 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:46.212304 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 07:22:46.212394 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:22:46.212500 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 07:22:46.212601 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.203888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.3.246.125:46779)
0924 07:22:46.203931 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.203933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.203940 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:22:46.203963 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.203964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.203974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.204074 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.204474 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.204981 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.205805 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.206033 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.206138 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.208907 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.208909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.208909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.208916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.211423 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.211790 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.211791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.211791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.211853 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.212058 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.212059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.212060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.212182 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.212262 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.212420 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.212500 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:22:46.213274 26733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:22:46.213616 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.213513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:34473 (local address 127.0.0.1:42974)
0924 07:22:46.213567 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.213596 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 07:22:46.213951 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.213863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:34473 (local address 127.0.0.1:42990)
0924 07:22:46.213907 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.213930 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 07:22:46.214040 26733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:34473: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
W20260924 07:22:46.239398 26815 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:22:46.239635 26815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:22:46.239665 26815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:22:46.239720 26815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:22:46.240908 26815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:22:46.240967 26815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:22:46.240990 26815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:22:46.241019 26815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:22:46.241042 26815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:22:46.242589 26815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.246.84:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.246.126:46591,127.3.246.125:46779,127.3.246.124:34473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/krb5kdc/kudu_127.3.246.124.keytab
--principal=kudu/127.3.246.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.246.124:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.246.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:22:46.242847 26815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:22:46.243074 26815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:22:46.245182 26733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } attempt: 1
I20260924 07:22:46.245594 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.245471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:34473 (local address 127.0.0.1:43002)
0924 07:22:46.245528 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.245566 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 07:22:46.245963 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.245866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:34473 (local address 127.0.0.1:43006)
0924 07:22:46.245912 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.245939 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 07:22:46.246048 26733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:34473: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
W20260924 07:22:46.247805 26820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:22:46.247850 26821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:46.247997 26815 server_base.cc:1034] running on GCE node
W20260924 07:22:46.247798 26823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:22:46.248306 26815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:22:46.248536 26815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:22:46.249713 26815 hybrid_clock.cc:648] HybridClock initialized: now 1790234566249697 us; error 41 us; skew 500 ppm
Sep 24 07:22:46 dist-test-slave-rc1w krb5kdc[26652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:22:46.252828 26815 init.cc:377] Logged in from keytab as kudu/127.3.246.124@KRBTEST.COM (short username kudu)
I20260924 07:22:46.253234 26815 webserver.cc:460] Webserver started at http://127.3.246.124:40565/ using document root <none> and password file <none>
I20260924 07:22:46.253463 26815 fs_manager.cc:359] Metadata directory not provided
I20260924 07:22:46.253505 26815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:22:46.253715 26815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:22:46.254763 26815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/data/instance:
uuid: "7495a71aa54e46e0b7ca86efa92a18d2"
format_stamp: "Formatted at 2026-09-24 07:22:46 on dist-test-slave-rc1w"
server_key: "fe150983152f42d0a90b6bc8514b99a2"
server_key_iv: "5f63d661a4bad6c8aff4ecc60ba2886e"
server_key_version: "encryptionkey@0"
I20260924 07:22:46.255122 26815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/wal/instance:
uuid: "7495a71aa54e46e0b7ca86efa92a18d2"
format_stamp: "Formatted at 2026-09-24 07:22:46 on dist-test-slave-rc1w"
server_key: "fe150983152f42d0a90b6bc8514b99a2"
server_key_iv: "5f63d661a4bad6c8aff4ecc60ba2886e"
server_key_version: "encryptionkey@0"
I20260924 07:22:46.256417 26815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:22:46.257104 26830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:22:46.257263 26815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:22:46.257359 26815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/data,/tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/wal
uuid: "7495a71aa54e46e0b7ca86efa92a18d2"
format_stamp: "Formatted at 2026-09-24 07:22:46 on dist-test-slave-rc1w"
server_key: "fe150983152f42d0a90b6bc8514b99a2"
server_key_iv: "5f63d661a4bad6c8aff4ecc60ba2886e"
server_key_version: "encryptionkey@0"
I20260924 07:22:46.257438 26815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:22:46.271198 26808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } attempt: 2
I20260924 07:22:46.271662 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.271534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:34473 (local address 127.0.0.1:43014)
0924 07:22:46.271592 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.271631 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 07:22:46.272024 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.271943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:34473 (local address 127.0.0.1:43030)
0924 07:22:46.271984 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.272007 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 07:22:46.272151 26808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.246.124:34473: Network error: Client connection negotiation failed: client connection to 127.3.246.124:34473: connect: Connection refused (error 111)
I20260924 07:22:46.274787 26815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:22:46.278047 26815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:22:46.278198 26815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:22:46.282187 26815 rpc_server.cc:307] RPC server started. Bound to: 127.3.246.124:34473
I20260924 07:22:46.282249 26882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.246.124:34473 every 8 connection(s)
I20260924 07:22:46.282577 26815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/data/info.pb
I20260924 07:22:46.283591 26883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:22:46.285095 26883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:22:46 dist-test-slave-rc1w krb5kdc[26652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.124@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
I20260924 07:22:46.291265 4057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwu82JT/build/release/bin/kudu as pid 26815
I20260924 07:22:46.291381 4057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwu82JT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234301618289-4057-0/minicluster-data/master-2/wal/instance
I20260924 07:22:46.291597 4057 external_mini_cluster.cc:1428] Setting key d43f23a93f0568fa832141e27b61b388
I20260924 07:22:46.296330 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.285821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.3.246.126:46591)
0924 07:22:46.285885 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.285887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.286045 (+ 158us) server_negotiation.cc:407] Connection header received
0924 07:22:46.286495 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.286496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.286506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.286569 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.287151 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.287620 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.288608 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.288792 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.288845 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.291890 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.291892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.291892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.291901 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.294812 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.295277 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.295279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.295280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.295347 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.295625 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.295627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.295627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.295768 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.295851 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.296142 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.296234 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:22:46.296504 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.285787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.126:46591 (local address 127.0.0.1:48072)
0924 07:22:46.285934 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.285974 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:46.286409 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:46.286621 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:46.286624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:46.286885 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:46.287069 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.287073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.287702 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.287703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.288533 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.288535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.288878 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.288879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.288977 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.290247 (+ 1270us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:46.290255 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:46.291778 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:46.294919 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.294927 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.294929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.295172 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.295421 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.295422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.295422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.295547 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.295843 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:46.295849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:46.295992 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 07:22:46.296117 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 07:22:46.296268 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260924 07:22:46.297137 26883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:22:46 dist-test-slave-rc1w krb5kdc[26652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.124@KRBTEST.COM
Sep 24 07:22:46 dist-test-slave-rc1w krb5kdc[26652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.124@KRBTEST.COM for kudu/127.3.246.125@KRBTEST.COM
I20260924 07:22:46.303563 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.291957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.3.246.124:34473)
0924 07:22:46.292175 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.292180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.292200 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:22:46.292279 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.292281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.292292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.292394 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.293212 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.294379 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.295144 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.295453 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.295535 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.298805 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.298812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.298812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.298818 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.302236 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.302633 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.302637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.302637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.302704 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.302902 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.302903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.302903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.303098 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.303303 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.303353 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.303437 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":125,"threads_started":1}
I20260924 07:22:46.307220 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.297424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.125:46779 (local address 127.0.0.1:51120)
0924 07:22:46.297491 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.297501 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:46.297571 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:46.297726 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:46.297726 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:46.297866 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:46.297914 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.297915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.298662 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.298663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.299297 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.299299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.299810 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.299811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.299871 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.301162 (+ 1291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:46.301167 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:46.302565 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:46.306061 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.306062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.306062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.306234 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.306532 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.306533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.306534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.306603 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.306884 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:46.306885 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:46.306929 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 07:22:46.307025 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 07:22:46.307128 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 07:22:46.307238 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.297510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.3.246.125:46779)
0924 07:22:46.297580 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.297582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.297593 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:22:46.297634 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.297635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.297646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.297722 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.298016 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.298548 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.299385 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.299727 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.299803 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.302680 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.302682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.302682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.302689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.305978 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.306365 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.306366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.306367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.306444 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.306682 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.306683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.306683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.306809 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.306892 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.307072 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.307148 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:22:46.307696 26883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:22:46 dist-test-slave-rc1w krb5kdc[26652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.126@KRBTEST.COM
Sep 24 07:22:46 dist-test-slave-rc1w krb5kdc[26652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.124@KRBTEST.COM for kudu/127.3.246.124@KRBTEST.COM
I20260924 07:22:46.315006 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.304354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.3.246.126:46591)
0924 07:22:46.304388 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.304390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.304398 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:22:46.304467 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.304468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.304475 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.304515 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.305135 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.305972 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.306960 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.307162 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.307248 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.310862 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.310863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.310864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.310871 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.313773 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.314156 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.314157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.314157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.314215 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.314410 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.314411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.314411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.314559 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.314738 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.314804 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.314897 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 07:22:46.317195 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.308200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:34473 (local address 127.0.0.1:43060)
0924 07:22:46.308308 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.308317 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:46.308363 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:46.308489 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:46.308490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:46.308613 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:46.308663 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.308664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.309291 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.309292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.310008 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.310011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.310488 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.310489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.310543 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.311671 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:46.311676 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:46.313050 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:46.316315 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.316317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.316317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.316532 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.316718 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.316718 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.316719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.316772 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.316999 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:46.316999 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:46.317031 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 07:22:46.317081 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 07:22:46.317142 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 07:22:46.317282 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.308269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.3.246.124:34473)
0924 07:22:46.308306 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.308308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.308356 (+ 48us) server_negotiation.cc:407] Connection header received
0924 07:22:46.308408 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.308409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.308418 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.308465 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.308744 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.309201 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.310101 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.310421 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.310490 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.313135 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.313136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.313137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.313144 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.316240 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.316599 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.316599 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.316600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.316656 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.316822 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.316823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.316823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.316937 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.316997 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.317119 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.317202 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":19}
I20260924 07:22:46.318588 26883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2: Bootstrap starting.
I20260924 07:22:46.319415 26883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:46.319697 26883 log.cc:824] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:46.320446 26883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2: No bootstrap required, opened a new log
Sep 24 07:22:46 dist-test-slave-rc1w krb5kdc[26652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.246.125@KRBTEST.COM
I20260924 07:22:46.321188 26733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } attempt: 2
I20260924 07:22:46.322688 26883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } }
I20260924 07:22:46.322835 26883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7495a71aa54e46e0b7ca86efa92a18d2, State: Initialized, Role: FOLLOWER
I20260924 07:22:46.322925 26883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } }
I20260924 07:22:46.323217 26889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } } }
I20260924 07:22:46.323307 26889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:22:46.323308 26883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 07:22:46.325107 26900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:22:46.325160 26900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:22:46.327236 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.315892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51126 (local address 127.3.246.125:46779)
0924 07:22:46.315928 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.315930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.315943 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:22:46.315968 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.315968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.315977 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.316050 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.316593 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.317379 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.318141 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.318360 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.318444 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.321417 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.321418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.321419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.321426 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.324551 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.325290 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.325291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.325291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.325354 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.325841 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.325842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.325842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.326423 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.326606 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.327068 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.327145 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 07:22:46 dist-test-slave-rc1w krb5kdc[26652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.126@KRBTEST.COM for kudu/127.3.246.124@KRBTEST.COM
I20260924 07:22:46.325318 26815 master_runner.cc:186] Error getting master registration for 127.3.246.126:46591: OK, instance_id { permanent_uuid: "287323bc23544c0297e260e285541a2b" instance_seqno: 1790234565974896 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:22:46.327790 26815 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:22:46.328989 4057 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:22:46.330180 26707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "287323bc23544c0297e260e285541a2b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48076
I20260924 07:22:46.332258 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.321447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:34473 (local address 127.0.0.1:43066)
0924 07:22:46.321493 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.321502 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:46.321544 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:46.321993 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:46.321994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:46.322150 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:46.322194 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.322196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.324064 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.324065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.324899 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.324902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.325338 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.325339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.325394 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.326457 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:46.326464 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:46.327810 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:46.331115 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.331116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.331117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.331366 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.331584 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.331584 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.331585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.331647 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.331954 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:46.331955 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:46.332000 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 07:22:46.332066 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:22:46.332157 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 07:22:46.332266 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.321500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.3.246.124:34473)
0924 07:22:46.321540 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.321542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.321550 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:22:46.321577 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.321578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.321587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.321967 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.322513 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.323965 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.324987 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.325169 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.325251 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.327894 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.327895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.327895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.327902 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.331006 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.331449 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.331450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.331451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.331512 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.331717 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.331718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.331718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.331883 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.331956 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.332105 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.332182 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:22:46.333933 26733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b: Bootstrap starting.
I20260924 07:22:46.334677 26733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:46.334990 26733 log.cc:824] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:46.335894 26733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b: No bootstrap required, opened a new log
I20260924 07:22:46.337248 26733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } }
I20260924 07:22:46.337391 26733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287323bc23544c0297e260e285541a2b, State: Initialized, Role: FOLLOWER
I20260924 07:22:46.337476 26733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } }
I20260924 07:22:46.337811 26733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:46.337834 26707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 07:22:46.337770 26903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } } }
I20260924 07:22:46.337893 26707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:46.337916 26903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:22:46.338637 26707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } }
I20260924 07:22:46.338894 26707 leader_election.cc:288] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [CANDIDATE]: Term 1 election: Requested vote from peers c6e32a14af484c5bb1e2ab3933b85b25 (127.3.246.125:46779), 7495a71aa54e46e0b7ca86efa92a18d2 (127.3.246.124:34473)
I20260924 07:22:46.339187 26857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "287323bc23544c0297e260e285541a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7495a71aa54e46e0b7ca86efa92a18d2"
I20260924 07:22:46.339306 26857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:46.340008 26857 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287323bc23544c0297e260e285541a2b in term 1.
I20260924 07:22:46.340234 26683 leader_election.cc:302] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287323bc23544c0297e260e285541a2b, 7495a71aa54e46e0b7ca86efa92a18d2; no voters:
I20260924 07:22:46.340355 26903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:22:46.340520 26903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [term 1 LEADER]: Becoming Leader. State: Replica: 287323bc23544c0297e260e285541a2b, State: Running, Role: LEADER
I20260924 07:22:46.340598 26903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } }
I20260924 07:22:46.341004 26903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 287323bc23544c0297e260e285541a2b. Latest consensus state: current_term: 1 leader_uuid: "287323bc23544c0297e260e285541a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } } }
I20260924 07:22:46.341080 26903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:46.341831 26665 master_runner.cc:186] Error getting master registration for 127.3.246.125:46779: OK, instance_id { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" instance_seqno: 1790234566124664 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:22:46.344669 26665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:22:46.347514 26922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:22:46.348012 26922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:22:46.353340 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.342447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.3.246.125:46779)
0924 07:22:46.342494 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.342495 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.342692 (+ 197us) server_negotiation.cc:407] Connection header received
0924 07:22:46.342832 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.342833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.342847 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.342908 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.343595 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.344311 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.345325 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.345689 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.345763 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.347966 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.347967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.347968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.347974 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.351798 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.352205 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.352206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.352207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.352265 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.352447 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.352448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.352448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.352573 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.353122 (+ 549us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.353163 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.353241 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 07:22:46.356659 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.341492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.3.246.126:46591)
0924 07:22:46.341532 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.341535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.344890 (+ 3355us) server_negotiation.cc:407] Connection header received
0924 07:22:46.344997 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.344997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.345009 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.345071 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.345775 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.346473 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.347476 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.347752 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.347820 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.351735 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.351737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.351738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.351745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.354783 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.355471 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.355484 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.355484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.355547 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.355859 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.355872 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.355873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.356000 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.356077 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.356482 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.356566 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:22:46.361119 26857 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2 [term 1 FOLLOWER]: Refusing update from remote peer 287323bc23544c0297e260e285541a2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:46.361515 26915 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:46.363253 26808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } attempt: 3
I20260924 07:22:46.363940 26889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 287323bc23544c0297e260e285541a2b. Latest consensus state: current_term: 1 leader_uuid: "287323bc23544c0297e260e285541a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } } }
I20260924 07:22:46.364038 26889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:22:46.364872 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.344804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.3.246.124:34473)
0924 07:22:46.344840 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.344842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.349106 (+ 4264us) server_negotiation.cc:407] Connection header received
0924 07:22:46.349219 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.349220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.349230 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.349259 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.355054 (+ 5795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.355825 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.356909 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.357180 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.357227 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.359708 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.359709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.359710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.359717 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.363013 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.363758 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.363759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.363760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.363823 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.364262 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.364264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.364264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.364408 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.364494 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.364679 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.364775 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
Sep 24 07:22:46 dist-test-slave-rc1w krb5kdc[26652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.246.125@KRBTEST.COM for kudu/127.3.246.124@KRBTEST.COM
I20260924 07:22:46.371137 26926 mvcc.cc:205] Tried to move back new op lower bound from 7332800783811588096 to 7332800783731732480. Current Snapshot: MvccSnapshot[applied={T|T < 7332800783811588096}]
I20260924 07:22:46.371606 26903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "287323bc23544c0297e260e285541a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } } }
I20260924 07:22:46.371707 26903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [sys.catalog]: This master's current role is: LEADER
I20260924 07:22:46.372061 26889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "287323bc23544c0297e260e285541a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } } }
I20260924 07:22:46.372144 26889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:22:46.372503 26927 mvcc.cc:205] Tried to move back new op lower bound from 7332800783811588096 to 7332800783731732480. Current Snapshot: MvccSnapshot[applied={T|T < 7332800783811588096}]
I20260924 07:22:46.373953 26922 catalog_manager.cc:1347] Generated new cluster ID: 2ece7be13412482ca2d23fac290d1fe9
I20260924 07:22:46.374008 26922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:22:46.379276 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.363494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.246.124:34473 (local address 127.0.0.1:43086)
0924 07:22:46.363572 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 07:22:46.363581 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:22:46.363617 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:22:46.365054 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:22:46.365055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:22:46.365224 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:22:46.365272 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.365273 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.366253 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.366255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.367101 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:22:46.367103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.367539 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.367540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:22:46.367596 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.368781 (+ 1185us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:22:46.368787 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:22:46.370307 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:22:46.377829 (+ 7522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.377830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.377830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.378113 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.378436 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:22:46.378437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:22:46.378437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:22:46.378516 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:22:46.378979 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:22:46.378980 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:22:46.379029 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 07:22:46.379075 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 07:22:46.379178 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 07:22:46.380177 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:46.363696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.3.246.124:34473)
0924 07:22:46.364937 (+ 1241us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.364939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.364946 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:22:46.364965 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.364966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.364977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.365041 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.365362 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.366131 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.367178 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:46.367477 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:46.367539 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:46.370401 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:22:46.370403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:22:46.370403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:22:46.370410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:22:46.375040 (+ 4630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.378202 (+ 3162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.378203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.378204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.378326 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:22:46.378601 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:22:46.378602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:22:46.378602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:22:46.378863 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:22:46.379781 (+ 918us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:46.379855 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 07:22:46.379994 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1225,"thread_start_us":73,"threads_started":1}
I20260924 07:22:46.381999 26808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25: Bootstrap starting.
I20260924 07:22:46.382725 26808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25: Neither blocks nor log segments found. Creating new log.
I20260924 07:22:46.383020 26808 log.cc:824] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25: Log is configured to *not* fsync() on all Append() calls
I20260924 07:22:46.389765 26922 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:22:46.390341 26808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25: No bootstrap required, opened a new log
I20260924 07:22:46.390476 26922 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:22:46.392356 26808 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } }
I20260924 07:22:46.392484 26808 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e32a14af484c5bb1e2ab3933b85b25, State: Initialized, Role: FOLLOWER
I20260924 07:22:46.392565 26808 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } }
I20260924 07:22:46.392907 26808 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:22:46.393926 26781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:22:46.394841 26781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25 [term 1 FOLLOWER]: Refusing update from remote peer 287323bc23544c0297e260e285541a2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:22:46.395098 26782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "287323bc23544c0297e260e285541a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c6e32a14af484c5bb1e2ab3933b85b25"
I20260924 07:22:46.395165 26931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:22:46.395885 26782 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287323bc23544c0297e260e285541a2b in term 1.
I20260924 07:22:46.397289 26934 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "287323bc23544c0297e260e285541a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } } }
I20260924 07:22:46.397490 26934 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:22:46.397560 26934 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 287323bc23544c0297e260e285541a2b. Latest consensus state: current_term: 1 leader_uuid: "287323bc23544c0297e260e285541a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } } }
I20260924 07:22:46.397619 26934 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:22:46.397651 26934 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "287323bc23544c0297e260e285541a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } } }
I20260924 07:22:46.397713 26934 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:22:46.398414 26922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b: Generated new TSK 0
I20260924 07:22:46.398634 26922 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:22:46.398800 26949 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:22:46.398854 26949 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:22:46.402253 26941 mvcc.cc:205] Tried to move back new op lower bound from 7332800783926931456 to 7332800783731732480. Current Snapshot: MvccSnapshot[applied={T|T < 7332800783811588096}]
I20260924 07:22:46.403116 26931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "287323bc23544c0297e260e285541a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287323bc23544c0297e260e285541a2b" member_type: VOTER last_known_addr { host: "127.3.246.126" port: 46591 } } peers { permanent_uuid: "c6e32a14af484c5bb1e2ab3933b85b25" member_type: VOTER last_known_addr { host: "127.3.246.125" port: 46779 } } peers { permanent_uuid: "7495a71aa54e46e0b7ca86efa92a18d2" member_type: VOTER last_known_addr { host: "127.3.246.124" port: 34473 } } }
I20260924 07:22:46.403208 26931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287323bc23544c0297e260e285541a2b [sys.catalog]: This master's current role is: LEADER
W20260924 07:22:46.434603 26736 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:46.430026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.3.246.126:46591)
0924 07:22:46.430073 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.430075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.433880 (+ 3805us) server_negotiation.cc:407] Connection header received
0924 07:22:46.433994 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.433996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.434007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.434040 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.434526 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48104: BlockingRecv error: recv got EOF from 127.0.0.1:48104 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 07:22:46.435360 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:46.430262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51142 (local address 127.3.246.125:46779)
0924 07:22:46.430312 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.430314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.434856 (+ 4542us) server_negotiation.cc:407] Connection header received
0924 07:22:46.434946 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.434947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.434957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.434987 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.435299 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51142: BlockingRecv error: recv got EOF from 127.0.0.1:51142 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 07:22:46.436141 26887 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:46.430706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.3.246.124:34473)
0924 07:22:46.430745 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.430746 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.435636 (+ 4890us) server_negotiation.cc:407] Connection header received
0924 07:22:46.435733 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.435734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.435745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.435792 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.436081 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43098: BlockingRecv error: recv got EOF from 127.0.0.1:43098 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 07:22:46.438930 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:46.438133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.3.246.125:46779)
0924 07:22:46.438167 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.438169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.438386 (+ 217us) server_negotiation.cc:407] Connection header received
0924 07:22:46.438489 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.438490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.438508 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.438537 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.438887 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51150: BlockingRecv error: recv got EOF from 127.0.0.1:51150 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 07:22:46.442018 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:46.441093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51154 (local address 127.3.246.125:46779)
0924 07:22:46.441125 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.441126 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.441463 (+ 337us) server_negotiation.cc:407] Connection header received
0924 07:22:46.441570 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.441571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.441582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.441611 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.441977 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51154: BlockingRecv error: recv got EOF from 127.0.0.1:51154 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 07:22:46.445938 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:46.445161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51158 (local address 127.3.246.125:46779)
0924 07:22:46.445193 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.445195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.445421 (+ 226us) server_negotiation.cc:407] Connection header received
0924 07:22:46.445505 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.445506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.445516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.445542 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.445898 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51158: BlockingRecv error: recv got EOF from 127.0.0.1:51158 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 07:22:46.452147 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:46.451145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51166 (local address 127.3.246.125:46779)
0924 07:22:46.451182 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.451184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.451580 (+ 396us) server_negotiation.cc:407] Connection header received
0924 07:22:46.451684 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.451685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.451696 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.451726 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.452095 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51166: BlockingRecv error: recv got EOF from 127.0.0.1:51166 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 07:22:46.462420 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:46.461457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.3.246.125:46779)
0924 07:22:46.461493 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.461495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.461830 (+ 335us) server_negotiation.cc:407] Connection header received
0924 07:22:46.461943 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.461944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.461956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.461986 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.462355 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51172: BlockingRecv error: recv got EOF from 127.0.0.1:51172 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 07:22:46.480609 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:46.479683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.3.246.125:46779)
0924 07:22:46.479722 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.479724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.480019 (+ 295us) server_negotiation.cc:407] Connection header received
0924 07:22:46.480129 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.480130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.480142 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.480173 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.480553 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51184: BlockingRecv error: recv got EOF from 127.0.0.1:51184 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 07:22:46.514871 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:46.513965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.3.246.125:46779)
0924 07:22:46.514008 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.514010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.514285 (+ 275us) server_negotiation.cc:407] Connection header received
0924 07:22:46.514389 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.514390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.514400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.514430 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.514811 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51196: BlockingRecv error: recv got EOF from 127.0.0.1:51196 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 07:22:46.581369 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:46.580387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.3.246.125:46779)
0924 07:22:46.580460 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.580463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.580663 (+ 200us) server_negotiation.cc:407] Connection header received
0924 07:22:46.580779 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.580780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.580794 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.580878 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.581304 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51202: BlockingRecv error: recv got EOF from 127.0.0.1:51202 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 07:22:46.711692 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:46.710917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.3.246.125:46779)
0924 07:22:46.711006 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.711007 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.711091 (+ 84us) server_negotiation.cc:407] Connection header received
0924 07:22:46.711148 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.711149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.711159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.711225 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.711596 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51212: BlockingRecv error: recv got EOF from 127.0.0.1:51212 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 07:22:46.969585 26813 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:22:46.968854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.3.246.125:46779)
0924 07:22:46.968922 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:46.968924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:46.968969 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:22:46.969022 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:46.969023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:46.969035 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:46.969093 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:22:46.969513 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51228: BlockingRecv error: recv got EOF from 127.0.0.1:51228 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:22:47.325572 26900 catalog_manager.cc:1259] Loaded cluster ID: 2ece7be13412482ca2d23fac290d1fe9
I20260924 07:22:47.325644 26900 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2: loading cluster ID for follower catalog manager: success
I20260924 07:22:47.326494 26900 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2: acquiring CA information for follower catalog manager: success
I20260924 07:22:47.326798 26900 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7495a71aa54e46e0b7ca86efa92a18d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:22:47.399304 26949 catalog_manager.cc:1259] Loaded cluster ID: 2ece7be13412482ca2d23fac290d1fe9
I20260924 07:22:47.399379 26949 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25: loading cluster ID for follower catalog manager: success
I20260924 07:22:47.400206 26949 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25: acquiring CA information for follower catalog manager: success
I20260924 07:22:47.400516 26949 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c6e32a14af484c5bb1e2ab3933b85b25: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:22:47.485487 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:47.482736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51242 (local address 127.3.246.125:46779)
0924 07:22:47.482873 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:47.482876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:47.482934 (+ 58us) server_negotiation.cc:407] Connection header received
0924 07:22:47.483003 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:47.483004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:47.483075 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:47.483165 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:47.483335 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:47.483846 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:47.484634 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:47.484855 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:47.484930 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:47.485108 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:47.485226 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:47.485320 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:47.485364 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 07:22:47.485386 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260924 07:22:47.489846 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:47.487456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.3.246.124:34473)
0924 07:22:47.487571 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:47.487573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:47.487598 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:22:47.487640 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:47.487640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:47.487686 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:47.487770 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:47.487861 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:47.488347 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:47.489068 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:47.489281 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:47.489355 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:47.489502 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:47.489601 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:47.489699 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:47.489740 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 07:22:47.489759 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":47,"threads_started":1}
I20260924 07:22:47.494457 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:47.491693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43114 (local address 127.3.246.124:34473)
0924 07:22:47.491732 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:47.491735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:47.491977 (+ 242us) server_negotiation.cc:407] Connection header received
0924 07:22:47.492026 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:47.492027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:47.492035 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:47.492111 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:47.492219 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:47.492890 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:47.493703 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:47.493977 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:47.494042 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:47.494213 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:47.494242 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:47.494308 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:47.494366 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 07:22:47.494387 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:22:47.494532 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:47.491391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.3.246.125:46779)
0924 07:22:47.491429 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:47.491431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:47.491621 (+ 190us) server_negotiation.cc:407] Connection header received
0924 07:22:47.491724 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:47.491725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:47.491738 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:47.491768 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:47.491964 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:47.492534 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:47.493626 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:47.493894 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:47.493942 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:47.494254 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:47.494284 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:47.494311 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:47.494414 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 07:22:47.494439 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:22:47.494858 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:22:47.491417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.3.246.126:46591)
0924 07:22:47.491514 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 07:22:47.491517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:22:47.491529 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:22:47.491654 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:22:47.491656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:22:47.491705 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:22:47.491779 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:22:47.491882 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:47.492695 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:47.493666 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:22:47.494027 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:22:47.494235 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:22:47.494304 (+ 69us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:22:47.494403 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:22:47.494740 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 07:22:47.494760 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 07:22:47.494787 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":55,"threads_started":1}
I20260924 07:22:47.506296 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 26665
I20260924 07:22:47.511541 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 26740
I20260924 07:22:47.516410 4057 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwu82JT/build/release/bin/kudu with pid 26815
2026-09-24T07:22:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2814 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5583 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (265905 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST